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Abstract 


MMOWGLI  (Massive  Multiplayer  Online  Wargame  Leveraging  the  Internet), 
sponsored  by  the  Office  of  Naval  Research,  is  an  online  game  platform  designed  to 
elicit  collective  intelligence  from  an  engaged  pool  of  world-wide  players.  In  the  past, 
the  Naval  Postgraduate  School  hosted  a  series  of  successful  games  including 
piracyMMOWGLI  (2011),  energyMMOWGLI  (2012)  and  biiMMOWGU{20^3)  which 
built  the  critical  mass  of  players  needed  to  find  creative  solutions  to  real-life,  difficult 
business  problems  such  as  piracy,  energy  and  business  innovation  initiatives  (bii). 
NPS  also  leveraged  MMOWGLI  with  the  analytic  framework  of  Lexical  Link  Analysis 
(LLA)  to  link  the  game  data  to  the  concepts  documented  in  two  business  processes  ( 
i.e.  improve  DoD  energy  efficiency  and  improve  future  open  systems  architecture 
[OSA]  strategy].  We  demonstrated  the  synergy  of  using  both  tools  to  gain  faster 
viability  of  new  ideas  to  improve  the  acquisition  process,  and  sorted  the  idea  cards 
that  might  be  good  candidates  for  further  investigation.  We  then  determined  that  the 
majority  of  Navy  programs  are  affected  by  (or  critically  dependent  on)  energy  issues, 
but  goals  and  terms  are  handled  inconsistently.  It  is  evident  that  MMOWGLI  together 
with  LLA  is  an  important  tool  for  comparing  and  considering  innovative  ideas  using 
social  media  games  to  improve  acquisition  processes. 

Keywords:  Massive  Multiplayer  Online  Wargame  Leveraging  the  Internet, 
MMOWGLI,  Collective  Intelligence,  Brainstorming  Social  Media,  Match  Matrix,  Idea 
Cards,  Action  Plans,  Open  Systems  Architecture,  OSA  Strategy,  Lexical  Link 
Analysis,  LLA,  Text  Mining,  Data  Mining,  Program  Elements,  Unstructured  Data, 
Data-Driven,  Acquisition  Process 
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Combining  MMOWGLI  Social  Media 
Brainstorming  With  Lexical  Link  Analysis  to 
Strengthen  the  DoD  Acquisition  Process 

Background 

Massive  multiplayer  online  wargame  leveraging  the  internet  (MMOWGLI), 
sponsored  by  the  Office  of  Naval  Research  (ONR),  is  an  online  game  platform 
designed  to  elicit  collective  intelligence  from  an  engaged  pool  of  world-wide  players, 
thus  invoking  a  fresh  approach  to  gather  data  from  a  targeted  community  via  crowd 
sourcing.  The  Naval  Postgraduate  School  (NPS)  is  the  primary  developer  of  this 
game  software.  In  the  past,  NPS  hosted  a  series  of  successful  games  including 
piracyMMOWGLI  (2011 -present,  ongoing),  energyMMOWGLI  (May  2012)  and 
biiMMOWGLI(business  innovation  initiative  MMOWGLI,  July  2013)  which  built  the 
critical  mass  of  players  needed  to  find  creative  solutions  to  real-life,  difficult 
problems  such  as  piracy  and  energy.  These  games  were  hosted  by  the  NPS 
Modeling  Virtual  Environments  and  Simulation  (MOVES)  Institute. 

We  leveraged  MMOWGLI  game  output  in  this  effort,  to  elicit  collective 
intelligence  from  the  acquisition  communities  for  two  business  processes: 

1 .  Improve  Department  of  Defense  (DoD)  energy  efficiency:  Studies 
evaluating  the  DoD’s  energy  use  have  been  conducted  by  the  Institute 
for  Defense  Analyses,  the  Defense  Science  Board  Energy  Security 
Task  Force,  and  JASON  (an  independent  scientific  advisory  group).  All 
three  studies  suggest  that  DoD  energy  inefficiency  is  a  significant 
liability,  a  constraint  on  operations  and  a  force-protection  challenge. 
More  specifically,  all  three  studies  led  to  two  consistently  held 
requirements  to  improve  DoD  energy  efficiency:  (1)  By  reducing 
energy  demand,  one  may  provide  operational  forces  greater  flexibility 
and  reduce  their  dependency  on  logistics  infrastructure,  and  (2)  the 
DoD’s  current  requirements  and  acquisition  processes  to  value  the 
technologies  with  the  potential  to  improve  energy  efficiency  (DoD 
Energy  Inefficiency,  2012). 


2.  Improve  open  systems  architecture  (OSA)  strategy:  The  assistant 
secretary  for  research  development  and  acquisition  (ASN  RDA) 
authorized  a  new  naval  OSA  strategy  in  November  2012  to  reduce  the 
total  ownership  cost  of  systems,  encourage  innovation,  and  more 
rapidly  deliver  needed  capabilities  to  the  warfighter.  This  strategy 
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specifically  challenges  the  naval  acquisition  workforce  to  institute 
measures  to  improve  competition,  eliminate  redundant  developments, 
and  coordinate  program  activities  that  promote  the  reuse  of  tactical 
products  across  sea  and  air  platforms.  The  acquisition  organization  is 
tasked  to  implement  the  strategy,  however,  success  will  require 
substantial  changes  in  the  Navy’s  business  practices,  organizational 
structures,  and  resource  planning. 

In  concert  with  the  updated  strategy,  Deputy  Assistant  Secretary  of  the  Navy 
(DASN)  -  Research,  Development,  Testing  &  Evaluation  (RDT&E)  created  a 
business  innovation  initiative  (Bll)  to  search  for  ways  to  overcome  the  inertia  many 
of  our  programs  of  record  (PoR’s)  suffer  today.  Mr.  Sean  Stackley  (as  cited  in 
Guertin,  Womble,  &  Bruhns,  2013),  the  ASN  RDA  said  in  a  recent  article: 

“The  value  of  an  innovation  initiative  is  to  explore  what  business-relationship 
changes  are  needed  to  open  up  competition;  incentivize  better  contractor 
performance;  increase  access  to  innovative  products  and  services  from  a  wider 
array  of  sources;  decrease  time  to  field  new  capabilities;  and  achieve  lower 
acquisition  and  life-cycle  costs  while  sustaining  fair  industry  profitability.”  (page  667). 

The  biiMMOWGLI  game  using  LLA  is  one  of  the  ways  to  achieve  these  goals. 
LLA  enables  the  graphic  depiction  and  quantitative  analysis  of  the  captured 
MMOWGLI  data,  as  explained  in  detail  in  the  business  innovation  initiative 
MMOWGLI  games  chapter.  We  reveal  the  new  knowledge  discovered  by  those 
participating  in  this  game  and  the  ideas  arising  from  the  data  linked  -  or  not  linked  - 
to  other  ideas,  or  perhaps  specific  guiding  documents.  We  are  thus  able  to  show 
relevance,  gaps,  and  consistency,  between  all  analyzed  data.  This  has  great 
ramifications  by  revealing  how  guidance  documents  may  be  missing  certain 
innovations,  or  how  they  might  show  acceptance  within  the  community.  We  show 
these  graphic  depictions,  and  their  supporting  match  matrices  in  later  chapters  and 
in  the  appendices. 

In  the  past  year,  we  applied  the  methodology  to  link  the  two  MMOWGLI 
games  to  the  concepts  documented  in  the  two  business  processes.  The  goal  of  this 
research  is  to  provide  an  innovative  platform  that  can  be  deployed  quickly  to 
mobilize  the  intellectual  capacities  of  the  research  and  professional  acquisition 
communities  to  provide  innovation  and  creative  ideas  to  address  the  challenges  and 
difficulties  in  the  two  business  processes.  We  also  compare  new  game  data  with  the 
most  recent  acquisition  data  and  measure  the  impacts  of  the  game  data  on  the 
current  state  of  the  policies  and  practices  in  a  broad  range  of  DoD  acquisition 
programs. 
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Methodology 

MMOWGLI  Game 


The  game  is  built  using  a  unique,  open  source,  software  adaptation  of  the 
Institute  for  the  Future  (IFTF)-designed  game  to  simulate  a  real-world  “brainstorm.” 

A  player  needs  to  register  with  a  required  game  ID  and  email;  the  last  name,  first 
name  and  other  personal  identification  information  (PM)  are  not  required. 

The  game  starts  with  an  explanation  of  the  situation  and  allows  a  player  to 
“Play  an  Idea”  or  “Take  Action.”  Players  can  then  choose  to  input  an  idea  or 
participate  in  the  discussion  of  an  existing  idea  in  the  categories  of  “Innovate”  and 
“Defend.”  The  discussion  can  be  in  one  of  five  categories:  expand — build  on  this 
idea  to  amply  the  impact;  counter — challenge  this  idea;  adapt — take  this  idea  in  a 
different  direction;  or  explore — something  missing.  Or  players  can  ask  a  question,  as 
shown  in  Figure  1.  In  the  end,  the  system  gathers  collective  intelligence  that  resides 
in  tree-structured,  color-coded  sets  of  ideas  and  discussions  in  text  format  as  shown 
in  Figure  2.  If  an  idea  and  its  associated  discussion  have  merit,  which  is  determined 
in  the  combination  of  the  player’s  score  and  the  Game  Master’s  recommendation,  it 
is  taken  into  a  separate  “Take  Action”  board  for  further  planning  and  deliberation. 


Figure  1 .  Categories  of  ideas  Based  on  the  Styles  of  Responses 
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Figure  2.  Ideas  Collected  in  the  Color-Coded,  Tree-Structured  Categories 

The  MMOWGLI  platform  is  suitable  for  tackling  a  broad  range  of  challenges 
for  national  security,  multiple  stakeholders,  and  small  or  large  communities  (e.g. 
corporations  and  research  communities  like  the  acquisition  system  communities).  It 
is  a  configurable  innovation  platform  that  can  be  adapted  to  any  scenario. 

Lexical  Link  Analysis  (LLA) 

As  in  military  operations,  where  the  term  situational  awareness  is  coined,  we 
note  that  that  our  efforts  can  inform  awareness  of  analyzed  data,  in  a  unique  way, 
that  help  improve  a  decision-makers’  understanding  or  awareness  of  the  data’s 
content.  We  therefore  define  awareness  as  the  cognitive  interface  between  decision 
makers  and  a  complex  system,  expressed  in  a  range  of  terms  or  features,  or  specific 
vocabulary  or  lexicon,  to  describe  the  attributes  and  surrounding  environment  of  the 
system.  Specifically,  LLA  is  a  form  of  text  mining  in  which  word  meanings 
represented  in  lexical  terms  (e.g.,  word  pairs)  can  be  represented  as  if  they  are  in  a 
community  of  a  word  network. 

Link  analysis  “discovers”  and  displays  a  network  of  word  pairs.  These  word 
pair  networks  are  characterized  by  one-,  two-,  or  three-word  themes.  Figure  3  shows 
a  visualization  of  common  lexical  links  shared  between  Systems  1  and  2,  shown  in 
the  red  box.  A  system,  or  a  corpus,  can  be  a  collection  of  documents  for  an  actual 
physical  system  (e.g.,  OSA  strategies,  ideas  in  a  MMOWGLI  game  or  simply  a 
category  of  information).  A  node  in  in  Figure  3  represents  a  word  in  a  corpus  and  a 
link  or  edge  represents  a  word  pair.  A  word  pair  is  a  bi-gram  (Manning  &  Schutze, 
1999)  word  pair  extracted  from  the  corpus.  Within  the  field  of  computational 
linguistics,  an  n-gram  is  a  sequence  of  n  items  matched  certain  probabilistic  patterns 
from  a  given  text.  Size  2  of  n-gram  is  a  bi-gram.  In  Figure  3,  each  color  of  a  link 
refers  to  the  collection  of  words,  lexicon  or  features  that  belongs  to  a  cluster  which 
describes  a  concept  or  theme.  In  overlapping  areas,  nodes  are  lexically  linked. 
Unlinked,  outer  vectors  (outside  the  red  box)  indicate  unique  system  features. 
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Figure  4  shows  the  information  from  three  categories  can  be  compared  and  Figure  5 
shows  the  information  from  two  time  periods  that  can  be  compared.  What  is  unique 
here  is  that  LLA  constructs  these  linkages  via  intelligent  agent  technology  using 
social  network  grouping  methods. 

The  closeness  of  the  systems  in  comparison  can  be  examined  visually  or 
using  the  quadratic  assignment  procedure  (QAP;  Hubert  &  Schultz,  1976  [  e.g.,  in 
UCINET];  Borgatti,  Everett,  &  Freeman,  2002)  to  compute  the  correlation  of  two  sets 
of  lexical  terms  from  two  systems  and  analyze  the  structural  differences  in  the  two 
systems  as  shown  in  Figure  6. 


Unique  Features 
System  i 


Unique  Features  of 
System  2 


Lexical  Links  between 
System  1  and  2 


Figure  3.  Comparing  Two  Systems  Using  LLA 


Figure  4.  Comparing  Three 
Categories 


Figure  5.  Comparing  Two  Time 
Periods 
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Figure  6.  QAP  Correlation  via  UCINET 

Figure  7  shows  a  visualization  of  LLA  with  connected  keywords  or  concepts 
as  clusters,  groups  or  themes.  Words  are  linked  as  word  pairs  that  appear  next  to 
each  other  in  the  original  documents.  Different  colors  indicate  different  clusters  of 
word  groups.  They  were  produced  using  a  social  network  community  detection 
method  (Girvan  &  Newman,  2002)  where  words  are  connected,  as  shown  in  a  single 
color,  as  if  they  are  in  a  social  community.  The  algorithm  clusters  the  words  into 
communities  based  on  the  word  pair  links  (edges)  among  the  words.  Traditional 
clustering  methods  typically  use  hierarchical  clustering  method  (Szekely  &  Rizzo, 
2005)  where  edges  with  strong  weights  progressing  towards  the  weakest  ones  are 
gradually  included  into  the  clusters.  Instead,  in  the  Girvan  &  Newman  method,  the 
communities  are  detected  by  progressively  removing  edges  that  are  least  central. 

For  example,  betweenness,  defined  as  the  number  of  shortest  paths  between  pairs 
of  nodes  that  run  through  a  node  (Freeman,  1977),  has  been  studied  in  the  past  as  a 
measure  of  the  centrality  of  nodes  in  networks.  The  edges  connecting  communities 
will  have  high  edge  betweenness.  By  removing  these  edges,  the  groups  are 
separated  from  one  another  and  so  the  underlying  community  structure  of  the 
network  is  revealed.  As  a  result,  a  word  center  is  formed  around  a  word  node 
connected  with  a  list  of  other  words  in  word  pairs.  For  instance.  Figure  8  shows  a 
detailed  view  of  a  theme  or  word  group  in  Figure  7.  The  center  words  are  “analysis, 
research,  approach.”  In  this  example,  we  use  three-word  such  as  “analysis, 
research,  approach”  to  label  such  a  group,  where  the  top-three  words  are  these  with 
the  highest  total  degree  of  centralities  (Freeman, 1979;  Wasserman  &  Faust,  1994). 
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Figure  7.  Word  and  Term  of  Themes  Discovered  and  Shown  in  Colored 

Groups 
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Figure  8.  A  Detailed  View  of  a  Theme  or  Word  Group  From  Figure  7 

The  detailed  steps  of  LLA  processing  include  the  following  steps: 

Step  1 :  Select  word  pairs  based  on  the  following  bi-gram  parameters: 

•  The  probability  threshold  for  one  word  next  to  another  word  in  a  word 
pair 

•  The  minimum  frequency  for  each  individual  word 

Step  2:  Apply  a  social  network  community  finding  algorithm,  i.e.  Newman  community 
detection  method  (Girvan  &  Newman  2002)  to  group  the  word  pairs  into  themes.  A 
theme  includes  a  cluster  of  lexical  word  pairs  connected  to  each  other. 

Step  3:  Compute  a  “weight,”  or  an  importance  measure,  for  a  theme. 

Step  4:  Sort  theme  weights  by  time,  and  study  the  distributions  of  the  themes  by 
time. 


The  outputs  of  LLA,  include  lexical  network  visualizations  such  as  the  ones  in 
Figure  3,  4,  5,  6,  7,  and  8,  radar  visualization,  and  matrix  visualization  (Zhao,  Gallup, 
&  MacKinnon,  2010).  The  word  pair  groups  or  themes  as  shown  Figure  7  and  8,  are 
further  divided  into  three  types  according  to  the  weights  in  Step  3: 

•  Popular  (P):  themes  containing  the  highest  number  of  mutually 
connected  word  pairs.  The  themes  represent  the  main  topics  in  a 
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corpus  at  the  time.  The  theme  represented  in  Figure  8  is  an  example  of 
a  popular  theme. 

•  Emerging  (E):  themes  containing  the  medium  number  of  mutually 
connected  word  pairs,  these  themes  may  grow  to  be  popular  over  time. 

•  Anomalous  (A):  themes  containing  the  lowest  number  of  mutually 
connected  word  pairs.  These  themes  may  be  off-topics  compared  to 
other  themes  and  may  be  interesting  for  further  investigation. 

Business  Problems  That  LLA  Addresses 

As  a  text  analysis  tool,  LLA  typically  addresses  the  business  problems  of 
discovering  themes  and  topics  in  the  unstructured  documents  and  sorting  the 
importance  of  the  themes  accordingly.  Current  methods,  for  example,  internet 
search  methods  of  ranking  pages,  require  established  hyperlinks,  citation  networks 
or  other  forms  of  crowd-sourced  collective  intelligence.  LLA  is  especially  useful  for 
the  data  without  hyperlinks  and  citation  networks,  for  example,  large-scale 
government  internal  documents.  Furthermore,  current  methods  typically  rank  the 
importance  of  the  information  based  on  their  popularity.  Instead,  we  found  that  in 
many  business  applications,  it  is  useful  to  rank  information  based  on  emerging 
importance  or  anomalousness. 

Current  research  of  social  network  analysis  mostly  focuses  on  people  or 
organizations  of  direct  associations  regardless  of  the  contents  linked.  The  so-called 
study  of  centrality  (Girvan  &  Newman,  2002;  Freeman,  1979)  has  been  a  focal  point 
for  the  social  network  structure  study.  Finding  the  centrality  of  a  network  lends 
insight  into  the  various  roles  and  groupings  such  as  the  connectors  (e.g.,  mavens, 
leaders,  bridges,  isolated  nodes),  the  clusters  (and  who  is  in  them),  the  network 
core,  and  its  periphery  (Orgnet,  2011). 

One  of  the  core  innovations  of  LLA  is  to  analyze  the  content  (e.g.,  documents 
and  social  media  communications)  created  by  social  entities  (e.g.,  people  or 
organizations),  therefore  create  alternative  networks,  i.e.  semantic  networks,  to  the 
traditional  social  networks.  The  resulting  networks  from  LLA  examine  both  social 
and  semantic  networks  in  terms  of  the  organizations  and  people  involved  in  the 
important  themes,  and  how  semantic  networks  might  suggest  improved  potential 
collaborations  and  predict  future  outcomes. 

Implementation  Details 

In  the  past  year,  we  continued  our  efforts  at  the  Naval  Postgraduate  School 
(NPS)  by  using  collaborative  learning  agents  (CLAs;  Ql,  2009)  and  expanded  to 
other  tools,  including  AutoMap  (Center  for  Computational  Analysis  of  Social  and 
Organizational  Systems  [CASOS],  2009)  for  improved  visualizations.  Results  from 
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these  efforts  arose  from  leveraging  intelligent  agent  technology  via  an  educational 
license  with  Quantum  Intelligence,  Inc.  CLA  is  a  computer-based  learning  agent,  or 
agent  collaboration,  capable  of  ingesting  and  processing  data  sources. 

We  have  been  generating  visualizations  including  a  lexical  network 
visualization  using  various  open  source  tools.  We  began  by  using  the  Organizational 
Risk  Assessment  (ORA;  CASOS,  2009)  tool  and  expanded  to  other  tools.  For 
example,  in  the  past  year,  we  developed  3D  network  views  using  Pajek 
(Batagelj,  Mrvar,  &  Zaversnik,  201 1 )  and  X3D  (Reid  201 1 ,  Brutzman  2008,  Web3D 
2013).  We  also  developed  our  visualizations  radar  view  and  match  matrix  view 
(Zhao,  Gallup,  &  MacKinnon,  2010). 

LLA  uses  a  computer-based  learning  agent  called  CLA  (Ql,  2009)  to  employ 
an  unsupervised  learning  process  that  separates  patterns  and  anomalies. 
Unsupervised  agent  learning  is  implemented  by  indexing  each  set  of  documents 
separately  and  in  parallel  using  multiple  learning  agents.  The  unsupervised  agents 
are  used  because  the  learning  data  for  supervised  agents  are  expensive  to  obtain. 
Multiple  agents  can  work  collaboratively  and  in  parallel.  We  set  up  a  cluster  utilizing 
Linux  servers  in  the  NPS  High  Performance  Computing  Center  (HPC)  to  handle  the 
large-scale  data  and  secure  environment  in  the  NPS  Secure  Technology  Battle 
Laboratory  (STBL). 

Relations  to  Other  Methods 

The  LLA  approach  is  more  properly  related  to  latent  semantic  analysis  (LSA) 
(Dumais,  Furnas,  Landauer,  &  Deerwester,  1988)  and  probabilistic  latent  semantic 
analysis  (PLSA;  Hofmann,  2000).  In  the  LSA  approach,  a  term-document  matrix  is 
the  starting  point  for  analysis.  The  elements  of  the  term-document  or  feature-object 
(term  as  feature  and  document  as  object)  matrix  are  the  occurrences  of  each  word  in 
a  particular  document,  i.e.  A  =  [aij],  where  aij  denotes  the  frequency  in  which  term  j 
occurs  in  document  /.  The  term-document  matrix  is  usually  sparse.  LSA  uses 
singular  value  decomposition  (SVD)  to  reduce  the  dimensionality  of  the  term- 
document  matrix.  SVD  cannot  be  applied  to  the  cases  where  the  vocabulary  (the 
unique  number  of  terms)  in  the  document  collection  is  large,  for  example,  the 
number  of  unique  terms  in  the  DoD’s  acquisition  documentation  approach  the  large 
value  that  would  make  SVD  inapplicable.  LSA  has  been  widely  used  to  improve 
information  indexing,  search/retrieval  and  text  categorization. 

A  recent  development  related  to  this  method  is  called  latent  Dirichlet 
allocation  (LDA;  Blei,  Ng,  &  Jordan,  2003),  which  is  a  generative  probabilistic  model 
of  a  corpus.  In  LDA,  a  document  is  considered  to  be  composed  of  a  collection  of 
words — a  “bag  of  words,”  where  word  order  and  grammar  are  not  considered 
important.  The  basic  idea  is  that  documents  are  represented  as  random  mixtures 
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over  latent  topics,  where  each  topic  is  characterized  by  a  statistical  distribution 
(Dirichlet  distribution)  over  the  corpus. 

Our  theme  generation  from  LLA  is  different  than  LDA,  in  which  a  collection  of 
lexical  terms  are  connected  to  each  other  semantically,  as  if  they  are  in  a  social 
community,  and  social  network  grouping  methods  are  used  to  group  the  words,  and 
unlike  LSA,  our  method  is  easily  scaled  to  analyze  a  large  vocabulary  and  is 
generalizable  to  any  sequential  data. 

LLA  is  further  related  to  tools  such  as  PageRank  (Brin  &  Page  1998; 
PageRank,  2013),  Automap  (CASOS,2009),  AlchemyAPI  (Al,  2013),  Semantica 
(SR,  2013)  for  entity  extraction,  text  analysis  and  sentiment  analysis,  WordNet 
(Miller,  1995),  and  Apache  Lucene(ASF,  2013),  OpenNLP(ASF,  2013),  and 
Mahout(ASF,2013),  with  the  best  of  each  incorporated  in  LLA. 

Anticipated  Benefits 

Our  LLA  method  provides  candidate  solutions  to  meet  the  critical  analytic 
needs  of  the  acquisition  research.  The  key  advantage  is  to  provide  an  innovative 
near  real-time  self-awareness  system  to  transfer  diversified  data  services  into 
strategic  decision-making  knowledge,  specifically  through: 

•  Automation:  High  correlation  of  LLA  results — with  the  link  analysis 
done  by  human  analysts — makes  it  possible  to  save  human  power  and 
improve  responsiveness.  Automation  is  achieved  via  computer 
program  or  software  agents  to  perform  LLA  frequently  -  and  in  near 
real-time. 

•  Discovery:  LLA  discovers  and  displays  a  network  of  word  pairs.  These 
word  pair  networks  are  characterized  by  one,  two  or  three  word 
themes.  The  weight  of  each  theme  is  determined  based  on  its 
frequency  of  occurrence.  It  may  also  discover  blind  spots  of  human 
analysis  that  are  caused  by  the  overwhelming  data  for  human  analysts 
to  consider. 

•  Validation:  LLA  may  provide  different  perspectives  of  links.  In  the 
acquisition  context,  links  discovered  by  human  analysts  may 
emphasize  component  and  part  connections  that  do  not  necessarily 
reflect  content  overlaps.  Consequently,  it  can  provide  improved  results 
in  terms  of  trust,  quality  of  association  discovery;  can  help  to  break 
through  different  levels  of  the  taxonomy  of  ignorance  (Denby  & 
Gammack,1999),  reach  across  organizational  boundaries,  and  help  to 
improve  organizational  reach. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  - 1 1  - 

NAVAL  POSTGRADUATE  SCHOOL 


other  Use  Cases 


In  this  section  we  discuss  other  recent  research  efforts  where  LLA  has  been 
implemented  to  uncover  meaning  and  depict  Big  Data  to  its  users. 

Discover  New  Knowledge  Using  Open  Social  Media  Data  Sources 

There  is  a  critical  need  for  Defense  Intelligence  Agency  (DIA)  to  discover  new 
sources  of  information  from  public  domains,  e.g.  from  various  social  media 
platforms,  and  then  link  them  with  intelligence  collected  for  other  intelligence 
applications.  We  demonstrated  how  LLA  can  be  applied  to  publically  available  social 
media  data  which  might  be  relevant  to  intelligence  applications.  We  develop  a 
specific  persona  archetype  and  to  analyze  all  available  data  derived  from  social 
media. 


Identification  of  NATO  Capabiiity  Requirements 

We  applied  LLA  to  analyze  the  documents  that  support  the  current  process  to 
identify  NATO  capability  and  force  requirements  from  the  current  process  and 
supporting  documents  to  help  determine  who  the  stakeholders  are,  i.e.  US  and 
Allied  organizations  involved  in  the  current  process,  in  an  effort  to  improve  EUCOM 
visibility  and  recommend  new  collaborations  toward  "Smart  Defense." 

DoD  Acquisition  Research  (Gallup,  MacKinnon,  Zhao,  Robey  &  Odell, 
2009;  Zhao,  Gallup  &  MacKinnon,  2010,  2011a,  2011b,  2011c,  2012a,  2012b, 
2013) 

The  US  DoD  acquisition  process  is  extremely  complex,  where  key  processes 
must  work  in  concert  to  deliver  the  capabilities  required  by  the  warfighters.  Each 
process  produces  a  large  amount  of  data  in  an  unstructured  manner.  There  has 
been  a  critical  need  for  automation,  validation,  and  discovery  to  help  acquisition 
professionals,  decision  makers  and  researchers  to  reveal  the  interrelationships 
among  the  data  elements  and  business  processes.  We  applied  LLA  to  extract  the 
links,  compare  the  trends  and  discover  previously  unknown  patterns  from  data  of 
three  armed-services  (Army,  Navy  and  Air  Force)  over  the  past  ten  years. 

Multi-Agency  Radiological  Responses  Plan  and  Exercise 

Every  year,  US  DHS  spends  large  amounts  of  money  to  conduct  training, 
exercises  and  simulations  to  prepare  for  emergency  responses.  These  exercises 
often  involve  processes  such  as  planning,  organizing,  directing,  and  monitoring 
activities  and  collaborations  of  multi-agencies.  The  activities  generate  large 
amounts  of  unstructured  data  for  sensemaking.  LLA  was  used  for  summarizing 
themes,  concepts  and  discovering  the  order  of  the  importance  of  the  events. 


-  12- 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Naval  Recruiting 

Facebook,  Twitter,  and  many  other  social  networking  sites  offer  virtual 
environments  for  meeting  possible  candidates  that  could  fit  service  entry  profiles. 
Sponsored  by  the  Navy  Recruiting  Command,  the  goal  of  this  project  was  to  collect 
and  match  large-scale  Facebook  public  fan  and  group  profiles  with  Navy-enlisted 
and  officer-rating  documents  to  improve  future  Navy  Recruiting  and  advertising 
efforts. 


Navy  Chief  of  Information  (CHINFO)  (Zhao,  Gallup,  &  MacKinnon,  2011a) 

The  case  study  involved  the  2006  U.S.  Coast  Guard  Live  Fire  case,  when  the 
Coast  Guard  planned  a  live  fire  training  program  in  the  Great  Lakes  area  in 
Michigan.  980  public  comments  and  200  pages  of  public  meeting  transcripts,  linking 
all  associated  comments,  and  then  generating  semantic  networks  over  time  by 
stakeholder  groups.  We  leveraged  LLA  to  determine  how  strategic  communications 
of  CHINFO  proliferate  through  various  open  sources. 

APAN  Network  and  Haiti  Operation  Data  Analysis  (Zhao,  MacKinnon,  & 
Gallup,  2012b) 

In  the  aftermath  of  the  Haiti  earthquake,  U.S.  military  and  civil  organizations 
provided  rapid  and  extensive  relief  operations.  LLA  was  used  to  analyze  trends  in 
interagency  synergy  from  data  collected  from  these  social  media  platforms  such  as 
Twitter,  Facebook,  news-feed  Web  sites,  official  PDF  briefing  documents,  situation 
reports,  forums  and  blogs  from  the  HAITI  HA/DR  Community  of  Interest  (COI)  on  the 
All  Partners  Access  Network  (APAN). 

Defense  Analysis 

Collecting  data  in  the  area  of  human  intelligence  (HUMINT),  we  performed  a 
feasibility  study  from  approximately  1500  reports.  Each  report  represented  a 
separate  event  including  post-blast  information,  and  after-action  reports  from  the 
Combined  Explosives  Exploitation  Cell  (CEXC)  and  data  from  other  reporting  tools 
used  in  Iraq  and  Afghanistan  war  activities  as  target  development,  civil  affairs, 
psychological  operations,  engagement,  or  indirect  fires.  Our  efforts  demonstrated 
the  capability  to  reconstruct  social  networks  of  people,  places,  and  events,  as  well 
as  to  reveal  trends  and  perhaps  predict  future  events. 

In  summary,  LLA  discovers  and  displays  these  networks  of  word  pairs  from 
large-scale  unstructured  data.  It  can  be  installed  as  a  search  and  knowledge 
management  tool  for  scoring  and  ranking  interesting  information  and  for  visualizing 
and  reporting  correlations  among  categories  and  layers  of  information  including 
social,  meta-data  and  semantic  links.  This  effort  then  presents  the  decision  maker 
with  previously  unavailable  and  emerging  patterns  and  themes,  as  well  as 
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unprecedented  levels  of  analysis,  thus  reducing  the  workload  and  overcoming  the 
blind  spots  of  human  analysts  and  providing  potential  automation.  For  example,  for 
the  recent  MMOWGLI  games,  LLA  was  leveraged  to  identify  potentially  interesting 
information  from  idea  card,  link  it,  then  recommend  them  for  action  plans  for  Game 
Masters. 

Figure  9  shows  a  MMOWGLI  game’s  content  and  attributes  can  be  processed 
into  the  inputs  (i.e.,  meta_data.txt  and  a  directory  of  text  files)  to  LLA. 


mmowgii 


M  IlfM  mmOWQll 


processXML.pl 
-  Output:  Meta_dSt^xt 


Figure  9.  Idea  Cards  Transformed  to  LLA  Inputs  (e.g.,  a  Directory  With  Files 
of  Content  of  Cards  and  Attributes,  and  meta_data.txt  to  LLA) 


Figure  10(a)  shows  word  pair  clusters  using  Newman  community  finding 
algorithm  (Girvan  &  Newman,  2002)  from  the  1®*  iteration.  Figure  10(b)  selected 
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lexical  terms  linked  to  the  most  central  nodes,  for  example,  “fuel,  shipboard,  liquid” 
from  the  2^’^  Iteration. 


JETTISONING* 


GEARED^ 

STANDARD^ 


BIOBUTANOL 


^ECRETE 

^LECT 

SYNTHETIC 


PROPULSION^ 
PAYLOADS  - 

SWATHS^  DEDICATED* 


VIBRATION^ 


SHIPBOARD, 


\/ 


.^EAVE 


^AIL 


- 

PRQPOSING^ 


AMENITIES  ^ 

CONSOLIDATE 

CONTAINMENT  4 
TASKS ^ 

TREAT 


45IDE 


:  0  67  nodeSj  73  links 


(a) 


(b) 


Figure  10.  Iterations  of  the  Two  Steps  LLA  Steps  Used  to  Group  Word  Pairs 

Into  Themes 


At  present,  LLA  computer  code  is  not  available  to  the  public  and  is  proprietary 
in  nature.  Dr.  Zhao  is  the  originator  of  the  software  code  which  was  used  in  support 
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of  numerous  government  projects  as  explained  above.  Future  efforts  might  include 
an  exportable  version  of  LLA. 

Research  Results 

We  applied  LLA  to  three  MMOWGLI  games,  specifically: 

•  energyMMOWGU  (May  2012):  560  players,  ~5000  idea  cards  and  68 
action  plans 

•  biiMMOWGLI  Round  1  (January  2013):  892  idea  cards,  11  action  plans 

•  biiMMOWGLI  Round  2  (July  2013):  2674  idea  cards,  15  action  plans 

From  these  games,  data  was  gathered  and  analyzed  by  LLA  to  show  the  correlation 
and  linkage  between  numerous  ideas  and  revealed  the  resulting  themes  as 
discussed  below. 

Energy  Game 

In  the  energyMMOWGU  game,  LLA  was  used  to  analyze  the  collected  data 
(idea  cards  and  action  plans)  retrieved  from  the  following  links: 

•  http://web.mmowgli.nps.edu/energy/ldeaCardChainEnergy2012.html 

•  http://web.mmowgli.nps.edu/energy/ActionPlanListEnergy2012.html 
The  LLA  was  performed  through  the  following  process: 

•  Prepare  acquisition  data.  Collate  key  terms  and  goal  statements  of 
current  acquisition  programs  within  the  congressional  budget 
processes  for  use  by  the  LLA  methodology 

•  Perform  link  analysis  and  correlation.  Compare  the  already- 
collected  energyMMOWGU  results  to  determine  action  plan  relevance 
on  a  program-by-program  basis 

As  shown  in  Figure  1 1 ,  our  goal  was  to  demonstrate  the  feasibility  of  the 
social  media  energyMMOWGU  game  as  an  innovation  platform  that  could  generate 
valuable  and  unexpected  contributions  and  solutions  for  improved  DoD  energy 
efficiency  through  the  acquisition  process,  by  linking  current  acquisition  programs 
with  the  energyMMOWGU  game  using  LLA.  We  achieved  this  objective  by 
performing  the  tasks  described  previously  and  detailed  in  the  next  section. 
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Acquisition 

Programs 

•Specific  products  & 
solutions  meeting 
warfighters' 
requirements 

•Levels  of  description 
details  embedded  in 
the  artifacts  in  the 
acquisition  process 
(e.g.,  PEs,  MDAPs, 
ACATTs) 

•Multiple  competing 
solutions  in  each 
program 

•Periodic  milestone 
reviews 


Lexical  Link  Analysis 
(LLA) 

•Ability  to  extract  key 
terms 

•Determine  cause  & 
effect  relations 
•Correlate  disparate  data 
sets  of  information 
•Screen  interesting 
information  that  might 
have  higher  targeted 
values 


Data  Mining  Results 

•Innovation  platform 
•Contributions/solutions 
towards  DoD  energy 
efficiency  via  the 
acquisition  process 


energyMMOWGLI 
•Social  media  game  for 
diverse  idea  generation 
and  planning 
•Focused  questions  for 
solving  the  DoD  energy 
inefficiency  relevant  to  the 
acquisition  process 
•Game 

engagement/participation 
■Contractors 
■Program  managers 
■Domain  experts  (e.g., 
symposium  attendees, 
faculties  &  students) 
■General  public 


Figure  11.  A  Glance  of  the  Proposal  Objective 


Prepare  Acquisition  Data 

The  goal  here  is  to  collate  key  terms  from  the  current  acquisition  program  in 
the  congressional  budget  process.  The  congressional  budget  process  documents 
e.g.  Program  Elements  [PEs]  from  http://www.dtic.mil/descriptivesum/  were  used  in 
this  task.  This  source  is  the  accurate  and  authoritative  high  level  of  artifacts  the  DoD 
RDT&E  process.  We  had  analyzed  part  of  these  documents  in  the  past  (Gallup, 
MacKinnon,  Zhao,  Robey  &  Odell,  2009;  Zhao,  Gallup  &  MacKinnon, 
2010,2011a,2011b,2011c,2012a,2012b,2013)  in  detail  using  the  LLA  method  jointly 
with  other  measures  such  as  cost,  schedule,  and  performance. 

Specifically,  we  collected  the  following  most  recent  (2013)  PEs  for  this  project: 


•  http://www.dtic.mil/descriptivesum/Y201 3_Navy.html 

•  http://www.dtic.mil/descriptivesum/Y201 3_AirForce.html 

•  http://www.dtic.mil/descriptivesum/Y2013_Army.html 
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Perform  Analysis  and  Correlation 

We  linked  the  energyMMOWGLI  data,  specifically,  38  action  plans  to  the  224 
Navy  PEs  to  evaluate  the  current  Navy  programs  relevant  to  the  game  data.  Figure 
12  illustrates  the  results  of  this  process  in  a  relevance  and  correlation  matrix. 


Figure  13(a)  shows  the  actual  match  matrix  sorted  with  Navy  PEs  (row)  that 
match  the  energyMMOWGLI  game  data  (column)  based  on  the  LLA  score.  Figure 
13(b)  shows  a  detailed  and  enlarged  part  of  Figure  13(a).  An  LLA  score  for  a  PE  is 
the  total  number  of  LLA  word  pairs  that  were  matched  with  the  game  action  plans. 
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Figure  13.  The  Overall  Match  Matrix  for  the  energyMMOWGLI  Game  Action 
Plans  and  2013  Navy  PEs;  (b)  Detail  of  Part  (a) 

The  top  five  most  relevant  PEs  from  Figure  7: 

•  PE  0603724N:  Navy  Energy  Program 

•  PE  0601 153N:  Defense  Research  Sciences 

•  PE  06021 23N:  Force  Protection  Applied  Res 

•  PE  0603573N:  Advanced  Surface  Machinery  Sys 

•  PE  0206624M:  Marine  Corps  Cmbt  Services  Supt 

In  the  actual  visualization  of  the  matrix,  one  is  able  to  click  on  the  online  link  for  the 
top  one  (PE  0603724N  in  Figure  13,  red  box)  leads  to  the  online  page  of  the  “Navy 
Energy  Program,”  which  is  an  overall  PE  specifically  focusing  on  Navy  energy  issues 
as  shown  in  Figure  14.  This  validates  that  the  LLA  extracted  the  relevant  keywords 
from  the  game  data. 
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UNCLASSIFIED 


Exhibit  R>2.  RDT&E  Budget  Item  Justification:  PB  2013  Navy 

DATE:  February  2012 

APPROPRIATION/BUDGET  ACTIVITY 

1319:  Research,  Development,  Test  &  Evaluation,  Navy 

BA  4:  Advanced  Component  Development  &  Prototypes  (ACD8P) 

R-1  ITEM  NOMENCLATURE 

PE  0603724N:  Navy  Energy  Program 

COST  ($  in  Millions) 

FY2013 

FY2011  FY2012  Base 

FY  2013  FY  2013 

OCO  Total  FY  2014 

FY  2015  FY  2016  FY  2017 

Cost  To 

Complete  Total  Cost 

Total  Program  Element 

33.124  70.538  55.324 

55.324  80.467 

93.031  52.278  53.272 

Continuing  Continuing 

0829.:  ENERGY  CONSERVATION 
(ADV) 

18.624  17.405  8.770 

8.770  10.865 

12.115  13.568  13.798 

Continuing  Continuing 

0838:  Mobility  Fuels  (ADV) 

10.520  15.888  11.071 

11.071  15.397 

14.537  12.004  12.280 

Continuing  Continuing 

0928:  Directed  Energy  Research 

13.404  16.243 

16.243  15.890 

19.482  2.869  2.930 

Continuing  Continuing 

0929;  Aircraft  Energy  Conservation 

23.841 

0.000  23.841 

0996:  Aircraft  Energy  Conservation 

19.240 

19.240  38.315 

46.897  23.837  24.264 

Continuing  Continuing 

9999:  Congressional  Adds 

3.980 

0.000  3.980 

A.  Mission  Description  and  Budget  Item  Justification 

This  program  supports  projects  to  evaluate,  adapt,  and  demonstrate  energy  related  technologies  for  Navy  aircraft  and  ship  operations  to:  (a)  increase  fuel-related 
weapons  systems  capabilities  such  as  range  and  time  on  station;  (b)  reduce  energy  costs;  (c)  apply  energy  technologies  that  improve  environmental  compliance;  (d) 
relax  restrictive  fuel  specification  requirements  to  reduce  cost  and  increase  availability  worldwide;  (e)  provide  guidance  to  fleet  operators  for  the  safe  use  of  commercial 
grade  or  off-specification  fuels  when  military  specification  fuels  are  unavailable  or  in  short  supply;  and  (0  make  needed  periodic  changes  to  fuel  specifications  to 
ensure  fuel  quality  and  avoid  fleet  operating  problems.  This  program  supports  the  achievement  of  legislated.  White  House,  Department  of  Defense,  and  Navy  Energy 
Management  Goals.  It  also  responds  to  direction  from  the  Office  of  the  Secretary  of  Defense,  the  Secretary  of  the  Navy,  and  the  Chief  of  Naval  Operations  to  make  up¬ 
front  investment  in  technologies  that  reduce  future  cost  of  operation  and  ownership  of  the  fleet  and  supporting  infrastructure. 

PE  0603724N:  Navy  Energy  Program 

Navy 

UNCLASSIFIED 

Page  1  of  35 

R-1  Line  #60 

Figure  14.  Navy  Energy  Program 


The  matrix  in  Figure  13  also  shows  a  holistic  picture  of  the  current  acquisition 
programs  in  connection  with  situations  in  which  DoD  is  energy  inefficient.  Directly 
looking  into  the  match  matrix,  as  illustrated  in  Figure  13,  can  be  overwhelming.  For 
that,  we  applied  LLA  to  discover  the  themes  and  divide  a  single  match  matrix  into 
many  match  matrices  with  different  themes  as  shown  in  Figure  15.  For  our  research, 
a  theme  is  a  network  or  community  of  word  pairs  that  are  related  to  each  other.  To 
discover  themes,  we  first  applied  LLA  to  compute  word  pair  clusters  using  the 
Newman  community  finding  algorithm,  in  which  equal  word  pairs  are  treated  as  if  in 
a  community  (Girvan  &  Newman,  2002).  Then  we  select  lexical  terms  linked  to  the 
most  central  nodes.  For  example  the  red  nodes  in  Figure  16  are  the  most  central 
nodes:  environmental,  ship  and  effective.  The  red  links  are  the  word  pairs  shared  by 
both  sources  (i.e.,  PEs  and  MMOWGLI  game  action  plans),  the  yellow  links  are  the 
word  pairs  unique  to  the  game  data,  and  the  green  links  are  those  unique  to  the 
PEs. 
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Figure  15.  Theme  Discovered  for  Navy  2013  PEs  Documents  and 
Energy  MMOWGLI  Data,  Sorted  According  to  Overlapping  Word 
Pairs  From  the  Two  Sources 
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Red  nodes;  the  most  "central"  nodes,  used  as  keywords  to  summarize  this  theme 
Red  Links:  shared  by  at  least  two  sources 

Green  Links:  from  Source  ARPM_mmowgli_energy 
Blue  links:  from  Source 


Figure  16.  Theme  395(E):  Environmental,  Ship  &  Effective 

A  separate  matrix  can  be  constructed  for  each  theme  including  the  links  of 
PEs  and  action  plans  with  the  word  pairs  that  only  belongs  to  the  theme.  In  Figure 
16,  the  correlation  matrix  for  Theme  395(E)  labeled  as  “environmental,  ship  & 
effective,”  has  the  highest  number  of  matched  word  pairs.  The  matched  PEs  are 
sorted  according  to  the  number  of  lexical  terms  matched  with  action  plans.  For 
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example,  the  top  matched  PE  is  “0603724N_PB_2013,”  titled  “Navy  Energy 
Program,”  which  indicates  that  this  is  a  current  Navy  program  dedicated  to  energy. 
We  used  this  matrix  to  determine  where  opportunities  reside  in  the  current  process 
to  include  energy-related  elements.  As  is  shown  in  Figure  17(a),  two  concepts, 
“energy  efficient”  (red  area  enlarged  in  Figure  17[b])  and  “ship  design”  9  (green  area 
enlarged  in  Figure  17[c])  are  dominant  in  this  theme.  They  are  dominant  since  four 
(action  17,  8,  18,  5  in  Figure  17[b])  and  two  (action  9  and  6  in  Figure  17[c])  out  of  38 
action  plans  contain  word  pairs  “energy  efficient”  and  “ship  design”  respectively. 

This  seems  to  suggest  that  “energy  efficient”  may  have  to  work  with  the  concept 
“ship  design.”  However,  among  the  12  PEs  that  mention  “ship  design”,  only  one 
mentions  “energy  efficient.”  (i.e.,  the  top  row  in  Figure  17[c],  corresponding  to  PE 
0603724N_PB_2013  -  the  Navy  Energy  Program).  This  indicates  there  is  a  gap,  or 
a  DoD  energy  inefficiency  area,  and  therefore,  an  opportunity  to  emphasize  the 
concept  “energy  efficient”  in  all  the  PEs  related  to  the  concept  “ship  design.” 
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Figure  17.  Match  Matrix  for  Theme  395  (E) 

Following  the  same  analysis,  Appendix  A  lists  more  gap  and  opportunity  areas 
discovered  by  LLA. 

Business  Innovation  Initiative  (Bll)  MMOWGLI  Game 

Round  1 

biiMMOWGLI  game  Round  1  was  performed  from  January  14,  2013  to 
January  15,  2013.  In  Round  1,  LLA  was  used  to  identify  potentially  interesting 
information  from  idea  cards  and  action  plans,  link  them  to  existing  business 
documents  and  show  their  interrelation  to  domain  experts.  We  performed  two 
separate  post-game  data  analyses. 

•  Idea  cards  (892)  and  action  plans  (1 1 )  were  compared  to  the  proposed 
OSA  strategy  (four  pages)  considered  by  players 

•  Idea  cards  (892)  and  action  plans  (11)  were  compared  to  the  OSA 
contract  guidebook  (158  pages)  familiar  to  most  players 

In  Round  1 ,  the  LLA  data  analysis  discovered  the  following: 
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•  Ideas  and  draft  action  plans  expressed  in  bii  game,  by  anonymous 
players,  showed  strong  consistency  with  the  concepts  in  the  Program 
Manager’s  Contract  Guidebook 

•  Metrics  indicate  the  draft  OSA  strategy  triggered  new  and  innovative 
ideas 

•  Metrics  did  not  indicate  that  the  OSA  strategy  was  risky,  controversial, 
impossible  to  implement  etc. 

LLA  also  discovered  eight  main  or  popular  themes,  reflecting  common  interest  of  the 
players,  using  the  following  keywords: 

•  Multiple  support  and  components 

•  Common  data,  data  model 

•  Component  reuse,  OSA 

•  Open  system  and  business 

•  Systems  architecture,  current  systems 

•  Specific  price  and  fee 

•  Existing  reusable  programs 

•  Engineering,  government  and  community 

We  also  found  that  innovative  ideas,  i.e.  gaps  between  the  game  data  and  the  OSA 
strategy  document,  in  the  following  areas  (themes)  listed  below: 

•  Small  and  shared 

•  Developed  and  built  faster 

•  Critical  definition 

•  Specific  price  and  fee 

•  Sponsors  change  and  risk 

•  Changing  requirements 

•  Interoperability  and  interfaces 

Figure  18  shows  one  example  theme  detailed  from  the  comparison  of  game 
data  with  the  OSA  strategy  document.  Red  nodes  show  the  top  three  word  hubs  with 
the  most  links  (or,  most  central).  Yellow  word  pairs  are  unique  to  action  plans,  green 
word  pairs  are  unique  to  idea  cards,  and  blue  word  pairs  are  unique  to  the  OSA 
strategy  document.  Red  word  pairs  are  found  in  more  than  two  sources. 
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Figure  18.  One  Theme  Matching  Keywords  Muitipie,  Support  and 

Components 


More  background  and  summary  for  Round  1  of  biiMMOWGLI  can  also  be 
found  in  (Guertin,  Womble,  &  Bruhns,  2013;  Zhao,  Brutzman,  &  MacKinnon,  2013). 

Round  2 

Round  2  of  the  biiMMOWGLI  game  was  conducted  between  from  July  15, 
2013,  to  July  31, 2013.  There  were  2674  idea  cards  and  15  action  plans  generated. 

In  Round  2,  we  applied  LLA  to  answer  the  business  question  we  started  to 
answer  in  Round  1:  specifically,  how  might  the  MMOWGLI  game  data  be  used  to 
improve  future  OSA  strategy?  We  also  aimed  to  answer  the  following  related 
questions: 

•  What  ideas  discussed  in  the  game  matched  with  the  OSA  strategy 
documents? 
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•  How  can  the  related  and  matched  ideas  be  used  in  a  way  that  is  useful 
for  future  OSA  strategies? 

To  answer  these  questions  in  detail,  in  Round  2,  we  focused  on  using  LLA  to 
produce  match  matrices  that  are  linked  to  the  new  OSA  strategy  document.  We 
then  divided  the  outputs  of  LLA  into  three  types  as  shown  in  Figure  19: 

•  Popularity  (P)  themes:  themes  containing  the  highest  number  of 
mutually  connected  word  pairs.  These  themes  represent  the  main 
topics  in  a  corpus  at  the  time. 

•  Emerging  (E)  themes:  themes  containing  the  medium  number  of 
mutually  connected  word  pairs.  These  themes  may  grow  to  become 
popular  over  time  as  we  show  later  in  the  examples. 

•  Anomaly  (A)  themes:  themes  containing  the  lowest  number  of  mutually 
connected  word  pairs.  These  themes  may  be  off-topics  compared  to 
other  topics  and  may  be  interesting  for  further  investigation. 
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82 

|75(A) 

69  [MMOWGLIJdeas 

68 

1  |EARLY,L’NIT,TESTS  ||(  A  i(  infovis  i|Q  l|a(  ds)  c  1  2  3  sunburst  pairs  hubs, 

76 

[all  [51  (A) 

68  [MMOWGLIJdeas 

68 

0  |PEOPLE.SOLLTIONS.COXTRACTTN'G  j|( .A i( infovis i|0  |aids!  c  1  2  3  sunburst  oairs  hubs 

77 

[all  |61(A) 

66  [MMOWGLIJdeas 

57 

6  |DATA,PROPRIET.ARY, MODELS  ||' A  if  infovis  i|3  i|aidsic  1  2  3  sunburst  pairs  hubs 

107 

[all  |14(A) 

65  [MMOWGLIJdeas 

63 

2  |PRACTICE,HUGE,\nSSION 

|(A  Kinfovis ;  |0  ||aCds  j  c  1  3  3  sunbursi  pairs  hubs; 

73 

[all  '|1  16(A) 

64  [MMOWGLIJdeas 

62 

1  |MAXAGEMEXT,EXSLTlE,COXTROL 

|f .A !( infovis >[1  |a(dsi  c  1  3  3  sunburst  oairs  hubs 

79 

[all  |52(A) 

62  [MMOWGLIJdeas 

60 

2  IIMPLEMEXTATIOX.EX.ABLES.COXTRACTORS  [cA  !i  infovis )  |0  ||a(  ds  >  c  1  3  3  sunburst  pairs  hubsi 

65  [ 

[all  |97(A) 

60  [MMOWGLIJdeas 

60 

0  |POST.STRATEGIC.EXPERIEXCE  A  xinfovis  1 10  ||aLdsi  c  1  2  3  sunburst  oairs  hubs; 

70  1 

[all  |127(A) 

58  [MMOWGLIJdeas 

55 

2  |EXGENEERIXG.EFF0RT.0A  iji  Ax  infovis  i[l  !|aLdu  c  1  3  3  sunburst  oairs  hubs 

76 

[all  |204(A) 

55  [MMOWGLIJdeas 

55 

0  |LOXG,TERM,COTS  1|! .A )t infovisi^[0  [at  ds :  c  1  2  3  sunburst  pairs  hubs' 

83 

[all  |1  62(A) 

46  [MMOWGLIJdeas 

45 

1  [TEAM.PROBLEM.CODE  j(  A  Xinfovis  i|0  ja(  ds  i  c  1  2  3  sunburst  oairs  hubs 

52 

[all  '|53(A) 

41  [MMOWGLIJdeas 

40 

1  |COXTRIBLTIOXS,FULLY,XML  !|f  A  xinfovis  i[0  i|a-ds-c  1  3  3  sunbursi  oairs  hubs^ 

45  [ 

[all  [21  (A) 

40  [MMOWGLIJdeas 

39 

1  |BASED,FUXCTTOX,TREATMEXT  i|cA  if  info^'is  i  |0  l|3(  ds)  c  1  3  3  sunburst  pairs  hubsj 

46  [ 

|197(A) 

40  [MMOWGLIJdeas 

36 

4  |SLTPORT.MEAXS,riTICALLY  ||i  A  Xinfovis  )[0  l|a(ds!cl  3  3  sunburst  oairs  hubs; 

42  1 

[all  |1  23(A) 

39  [MMOV^'GLIJdeas 

33 

5  |NEEDED.KX0\\XEDGE.RESEARCH  j;.Ax  infovis  j[l  'ja(  ds  i  c  1  3  3  sunburst  pairs  hubs 

43  1 

ALL  1183(A) 

36  IMMOWGLI  ideas 

36 

0  |REOUIREMEXT.ST.ART.E\'OL\TXG  f  A  i:  liifo'.  is  i  k)  lai  ds  i  c  1  3  3  sunburst  oairs  hubs 

38 

Figure  19.  Themes  of  Popularity  (P),  Emerging  (E)  and  Anomaly  (A) 
Discovered  Using  LLA  in  the  Round  2  idea  Cards 
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Figure  20  shows  the  detail  for  the  theme  centered  around  “Existing,  Future, 
Innovation.”  It  shows  the  contrast  between  what  is  only  in  the  OSA  strategy 
document  (green)  and  what  is  only  in  the  game  idea  cards  only  (yellow) .  It  also 
shows  overlap  (red)  in  these  two  data  sources. 


From  Figure  20,  we  see  that  the  game  generated  many  new  concepts  (yellow 
links)  centered  on  the  theme.  These  new  concept  (for  example,  “leveraging  existing,” 
“OSA  innovation,”  “incentivize  innovation,”  and  “future  supportability,”  etc.)  can  be 
used  to  improve  the  future  OSA  strategy  document.  Appendix  B  lists  the  top  themes 
in  Figure  19. 

Figure  21  shows  a  match  matrix  for  the  idea  cards  in  the  biiMMOWGLI  Round 
2  matched  with  the  OSA  strategy  document,  where  the  matched  word  pairs  belong 
to  the  category  “Popularity”.  This  category  includes  the  concepts  that  are  common 
knowledge  to  the  acquisition  community. 

In  Figure  21,  clicking  the  link  “open  standards”  opens  the  LLA  search  results 
shown  in  Figure  22,  which  identify  the  idea  cards  and  the  OSA  strategy  containing 
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the  word  pair  “open  standards.”  One  can  see  the  cards  enrich  the  concept  “open 
standards”  in  the  OSA  strategy  with  related  concepts  such  as  “giant  loyalty”  (card 
2547),  “future  roadmap”  (card  1062),  “common  playing  field”  (card  1739)  and  “open 
APIs”  (card  2612). 


Lexical  Links  Updated  on  Fri  Aug  30  09:49:38  2013  Using  'Popularits'  Word  Pairs 

Id  |MMO\VGLI ideas(Oiilme) 

MMOWGLI  ideas 

OS.AJrcchure-20 1 3reduced.pdf 

Total  Row  LLA  Score 

More  Links  1 

1  ICard  2063 

Card  2063 

396.00:INSERTIOX  C54P.ABILITY(202.00'i:M4XAGERS  PROG3L4:'.:i  194.00) 

396 

•) 

ICard  1087 

icard  1087 

388.00:PROGRESS  OSAri94.00'l:ASSESS  OSAf  194.001 

388 

3  |Card  1067 

|Card  1067 

202.00:.ARCHITECTURE  SYSTEMSr202.00) 

202 

indS - 

202.00:.4RCHITECTURE  SYSTEMSr2Q2.001 - 

TOT 

5 

IC'ard  913 

Card  913 

202.00:.^CHITECTURE  SVSTEMSC202.00) 

202 

6  tard  1414 

Card  1414 

202.00:ST.AXDARDS  QP£yf202.001 

202 

7 

ICard  2547 

Card  2547 

202.00:ST.AXDARDS  OPEXf202.001 

202 

s 

icard  1739 

Card  1739 

202.00;STAND.ARDS  OPEX(202.00) 

202 

p 

ICard  1062 

Card  1062 

202.00:ST.AX'D.ARDS  OPEXf202.001 

202 

10 

iCard  1701 

Card  1701 

202.00:ST.AX'D.ARDS  OPEXr202.00) 

202 

iT 

jCard..l060 

Card  1060 

202.00:STAXT).ARDS  QPEXf202.001 

202 

11 

|card  1954 - 

Card  1954 - 

202.00;STAXDARDS  QP£X(202.00) - 

202 - 

13 

jcard  1061 

Card  1061 

202.00:ST.YXD*YRDS  OPEXf202.00) 

202 

14 

icard  1126 

Card  1126 

202.00:IMPLE\EXTATIOXS  OSAf202.00'l 

202 

15 

jcard  1370 

Card  1379 

194.00:\LAXAGERS  PROGR.AMri94.00'l 

194 

16 

jcard  14S7 

Card  14S7 

194.00:MAXAGERS  PROGKAMf  194.001 

194 

17 

jcard  033 

Card  933 

194.00:MAXAGERS  PROGR^AMf  194.001 

194 

18 

jcard  1554 

Card  1554 

194.00:STR.4rEGY  OSAri94.00) 

194 

19 

jcard  917 

Card  917 

194.00:\L4XAGERS  PROGR.4Mri94.001 

194 

20 

ICard  2512 

Card  2512 

194.00:M4XAGERS  PROGR.4Mr 194.001 

194 

Figure  21 .  A  Match  Matrix  for  the  biiMMOWGLI  Game  Round  2  Cards 
Matched  With  the  OSA  Strategy  Document  Using  Popularity  Word 

Pairs 
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LLA  Search  Results 

’’STAXDAEDS  OPEX’  returned  ti  results 

Card  2.J^47.txt 

UH  THE  ONE  WHERE  APPLE  Avoms  USING  GOOD  TECH  TO  LOCK  IN 

LARGE  MARKET  SHARE  AND  CLAIM  GIANT  ROYALTIES  AUDIO  VIDEO  ETC 

http:  / /localhost:  8o8o/hii_2  /  publish/MMOWGLI_ideas/Card_2547.txt 

[20Q06.00  ~  6.oOjavoid£  apen^giant  myaltiffi^giant  clauu^avoids  apple^standards  opeUj) 

Card  iQ62.t?d: 

FOCUS  ON  FUTURE  ROADMAP  AND  MOST  VOLATILE  ELEMENTS  TO  ENSURE 
USED 

http:  / /iDcalhost:  8o8o/bii_2  /  publish/MMOWGLI_idea£/Card_io62  .txt 
[20004.00  ~  4.oOjStaQdards  apen^ensure  open,  roadmap  future,) 

Card  i7;^Q.txt 

USING  FOR  DATA  INTERCHANGE  PROVIDES  A  COMMON  PLAYING 

FIELD  FOR  MANY  SYSTEMS  TO  INTEROPERATE  ROYALTY  FREE 
http:  / /localhost:  8o8o/bii_2  /  publish/MMOWGLI_ideas/Card_i739.txt 
[20004.00  ~  4.oOjplaying  held, standards  open, playing  common,) 

OSABroc]niTe-20i.‘^reduced.pdf 

PROMOTE  TAILOR  ABLE  RELATIVE  TO  TRF  ATTRIBUTES 

http:  / /localhost:  8o8o/bu_2  /  publish/  OSA_strategy/ OSABrochure-2013reduoed.pdf 
(2  0004. 00  ~  4.00,  relative  standards, tailor  promote,standards  open,) 

Card  2g:84.txt 

AND  PUBLISHED  AND  APIS  MUST  BE  USED 

http:  / /localhost:  8o8o/bii_2  /  publish/StIMOWGLI_ideas/Card_2584.l3tt 
[20003.00  ~  3.oOjpublished  standards, apts  open,) 

Card  26i2.txt 

I  AGREE  WITH  THE  PUBLISHED  AND  APISCARD 

http:  / /localhost:  8o8o/bii_2/  publish/MMOWGlI_ideas/Card_25i2  .txt 
[20003.00  ~  3.oo,published  standards, apis  open,) 

Card  io6i.txt 

OSAIS  TOO  LOOSE  TO  DEVELOP  PRODUCT  LINES  NEED  TO  USE  CONSORTIUM  BASED 

http:  / /localhost:  8oSo/bii_2 /publish/MMOWGLI_ideas/Card_io6i  .Ixt 
[20003 .00  ~  3.oo,standards  opea,bas0d  open,) 


Figure  22.  LLA  Search  Results  for  “Standards  Open” 

Similarly,  in  Figure  23,  clicking  the  link  “life  cycle”  reveals  the  LLA  search 
results  shown  in  Figure  24,  and  identifies  the  cards  and  the  OSA  strategy  containing 
the  word  pair  “life  cycle”.  Additionally,  the  idea  cards  from  the  game  enrich  the 
concept  by  providing  linked  meanings  such  as  “life  cycle”  in  the  OSA  strategy  with 
related  concepts  such  as  “operational  scenario,”  “SE  development  ”  (card  2255), 
“sustainment  cost,”  “business  models”  (card  2300),  “automatic  maintenance,” 
“infrastructure  support”  (card  2467  marked  as  “super  interesting”  by  an  analyst). 
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“system  design”  (card  2481,2308),  “infrastructure  costs”  (card  2308)  and  “prohibit 
contracts”  (card  1223). 


Lexical  Links  Updated  on  Fri  Ang  30  09:59:05  2013  Using  ’Emerging’  Word  Pairs 


Id 

|MMOWGLI ideas(C)iiline) 

^IMOWGLIJdeas 

0S.43r 

chuft-2013rtfac;aDilf 

Total  Row  LLA  Score 

More  Links 

1 

|Card  19P: 

ICard  1995 

316.00:CYC2H  1IFE(158.00);1IFH  SYSTF:\:ri58.00) 

316 

•> 

ICard  2255 

|Card  2255 

316.00;CYCLE  lIFEfl58.00'):LIFE  SYSTEMf  158.00) 

316 

3 

[Card  2235 

|Card  2235 

316.00:0\\’NERSHIP  TOTALfl58.00'>:O\\’NERSHIP  COSTf  158.001 

3^ 

4 

|C-ard  2310 

kZ'ard  2310 

316.00:CYCLE  lIPEri58.00'l;LIFE  PROGR-YMC  158.00) 

316 

5 

jcai-d  2556 

K'ai-d  2556 

316.00;0\\’NERSHIP  COSTS(158.00);O\\’NERS:^  TOT.41(  158.00) 

316 

6  ICard  1340 

^‘ard  1340 

316.00;0\\'N'ERSHIP  COSTS(158.00):O\\'N'ERS?nP  TOTALf  158.00) 

316 

7 

|Ca:d  26S1 

t'ard  2681 

316.00:CYCLE  1IFE(158.00):SA\TXGS  COSTfl58.001 

316 

s 

|Card  2667 

jcard  2667 

246.00:FRAMEWORKS  TECHMCALfl23.001:COXSOLIDATE  TECHNIC ALf  123. 001 

246 

9 

IC’ard  14P: 

kZ'ard  1495 

158.00;SA\TNGS  COST(158.00) 

158 

10 

|Ca:d  1223 

kZ’ard  1223 

158.00:CYCLE  1IFE(158.00) 

158 

11 

^'ard  1198 

^•ard  1198 

158.00:SA\TXGS  COSTfl58.00) 

158 

n 

jcard  1600 

|Card  1600 

158.00:SA\TXGS  COSTf  158.00) 

158 

111 

jcard  1768 

ICard  1768 

158.00:SA\TXGS  COSTfl58.00) 

158 

14 

jcard  1598 

IC’ard  1598 

158.00;SA\TN'GS  COSTri58.00) 

158 

15 

ICard  1531 

ilCard  1531 

158.00:SA\TN’GS  COSTri58.00) 

158 

16 

jcard  1601 

ic-ard  1601 

158.00:SA\TXGS  COSTf 158.00) 

158 

17 

ICard  1945 

icard  1945 

158.00:CYCLE  LIFE(  158.00) 

158 

18 

jcard  2256 

|Card  2256 

158.00;LIFE  SYSTEM(  158.00) 

158 

19 

ICard  1017 

ic’ard  1017 

158.00;CYCLE  1IFE(158.00) 

158 

20 

jcard  2510 

iicaid  2510 

158.00;CYCLE  1IFE(158.00) 

158 

[21 

|Ca:d  1377.suoeiIiiLefesdn£ 

IlCard  1377.suoeiIiKe:esLiii£ 

158.00:SA\TXGS  COSTf 158.00) 

158 

jcard  1335 

ICard  1335 

158.00:SA\TXGS  COST(158.00) 

158 

23 

[Card  2050 

jcard  2050 

158.00;CYCLE  1IFE(  158.00) 

158 

24 

[Card  2467  sucerinteresdng 

[Card  2467.superlnterestin2 

158.00:CYCLE  LIFE(1 58.00) 

158 

25 

icard  1467 

ICard  1467 

158.00;SA\TN'GS  COSTf  158.00) 

158 

26 

jcaid  1150 

ICard  1150 

158.00:SAMXGS  COSTf 158.00) 

158 

27 

ICard  2481 

ICard  2481 

158.00:CYCLE  LIFE(  158.00) 

158 

28 

[Card  1232 

[Card  1232 

158.00:SA\TXGS  COSTf  158.00) 

158 

29 

jcard  1764 

|Card  1764 

158.00:SAMXGS  COSTfl58.00) 

158 

3011c  ard  1555 

jCard  1555 

158.00;SA\TXGS  COST(  158.00) 

158 

31 

ICai-d  1305 

icaid  1305 

158.00;SA\TS*GS  COST(158.00) 

158 

32 

|Card  2392 

|Card  2392 

158.00;SAMNGS  COST(158.00) 

158 

33 

ICard  1769 

icard  1769 

158.00:SA\TXGS  COSTfl58.00) 

158 

34 

icard  1538 

jcard  1538 

158.00:SA\TXGS  COSTf 158.00) 

158 

;35 

ICard  2458 

IC'ard  2458 

158.00:CYCLE  lIFEfl58.001 

158 

[36 

|Caid  963 

jcard  963 

158.00;SA\TXGS  COSTf  158.00) 

158 

37 

[Card  929 

jcard  929 

158.00:SAMXGS  COSTf  158.00) 

158 

l3 8 

jcard  1 '63 

jCard  1763 

158.00;SA-'IXGs  COsT(  158.00) 

158 

Figure  23.  A  Match  Matrix  for  the  biiMMOWGLI  Game  Round  2  Cards 
Matched  With  the  OSA  Strategy  Document  Using  Emerging  Word 

Pairs 
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LLA  Search  Results 

'CYCLE  LIFE"  returned  33  results 

Card  22f:f:.txt 

USE  OF  MODEL  EASED  SE  DEVELOPMENT  ENABLES  AUTOMATED  TESTING  AGAINST 
OPERATIONAL  SCENARIOS  ACROSS  SYSTEM 

http:  / /localhost:  Bo&o/bii_a/publi5hyM3^IOWGLI_idea£/  Card_2i255.txt 
[20007.00  ~  y.oojcycle  Ufe, scenarios  operationalise  developmentjUfe  systenijse 
based, enables  development,) 

Card  2.'^QQ.t?d: 

COST  HOW  CAN  WE  REQUIRE  CREATION  OF  LONG  TERM  MECHANISMS  AND 
BUSINESS  MODELS  FORSUPPORT  THAT  REDUCE  SUSTAINMENT  COST 
http:  / /localhost:  &o8o/bu_2/publish/MMOWGLI_ideas/  Card_2300.txt 
[20006.00  ~  B.oOjCycle  oostiSustainment  oostimodels  businesSimechanisms  termjCreation 
require,) 

Card  2467.superI11teresti11g.txt 

AUTOMATE  MAINTENANCE  AS  PART  OF  DESIGN  TO  DRAMATICALLY  REDUCE 
SUPPORT  INFRASTRUCTURE 

http:  / /localhost:  8oSo/bii_2/publish/MMOWGLI_ideas/  Card_2467.EuperInteresting.txt 
[20006.00  ~  6.oOjCyde  life, automate  maintenance, cycle  support, Ufe  reduoe,infrastructure 
support,) 

Card  248i.txt 

GIVE  CREDIT  TO  THE  SYSTEM  DESIGN  THAT  ADDRESSES  COSTS  AND  TECH 

REFRESH 

http:  / /locaIhost:8o8o/bii_2/publish/MMOWGlJ_ideas/Card_24Si  .txt 
[20006.00  ~  6. 00, addresses  ltfe,cycle  Ufe, cycle  co5ts,credit  give,desigu  system,) 

Card  2^Q8.txt 

COST  CAN  WE  PROPOSE  WAYS  TO  ELIMINATE  SUPPORT  AND 
INFRASTRUCTURE  COS'K  AS  PART  OF  TECHNICAL  PROPOSALS  FORSYSTEM  DESIGN 
http:  / /localhost:  &oSo/bU_2/publish/MMOWGLI_ideas/  Card_230S.txt 
[20006.00  ~  6.oOjinfrastructure  costs,cycle  cost, propose  ways,proposals  technical,design 
system,) 

Card  I223.txt 

PROHIBIT  CONTRACTS  THAT  INCLUDE  BOTH  SYSTEM  DEVELOPMENT  AND 
SUSTAINMENT 

http:  / /locaIhost:8o8o/bu_2/publLsh/MMOWGLI_ideas/Card_i  223  .tJtt 

[20005.00  ~  5.oo,cycle  Ufe,prohibit  contracts, sustainment  cycle,development  system,) 

Card  268l.txt 

EVALUATE  TOTAL  COSTS  INCLUDING  EEPAIRAND  MAINTENANCE 


Figure  24.  LLA  Search  Results  for  “Cycle  Life” 

In  Figure  23,  clicking  on  the  link  on  “cost  savings”  reveals  the  LLA  search 
results  shown  in  Figure  25,  and  identifies  the  ideas  cards  and  the  OSA  Strategy 
containing  the  word  pair  “cost  savings”.  In  this  instance,  the  idea  cards  enrich  the 
concept  “cost  savings”  in  the  OSA  strategy  with  related  concepts  such  as  “cost 
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influence,  incentive  plans  (card  1232),  evaluation  criteria,  CPARS  review,  future 
RFPS(card  1601),  source  selection(card  1467),  actual  cost,  FOSS  software, 
software  licenses,  contract  execution  (card  1484),  program  funds,  cost  realized, 
expanded  funds  (card  1495),  etc. 


LLA  Search  Results 

'■SxWTXGS  COST"  retUFQed  27  re£iilt& 

Card  I2,'^2.txt 

MIERORINDUSTRY  AMD  CREATE  INCENTIVE  PLANS  FOR  KEY  DECISION  MAKERS  WHO 

CAN  INFLUENCE  FOR  ACQUISITION  CONTRACTS 

http:  / /localhost:  8o8o/bii_2  j  publish/MMOWGLI_ideas/  Card_  1232.txt 

[aoooS.oo  ~  cost, mirror  industryjsavings  oost^acquMtion  contracts, create 

mcentive,decision  key^plans  incentive,} 

Card  i6Qi,txt 

RECOGNIZE  SAVING  AS  PART  OF  THE  CPARS  REVIEW  AND  WEIGHT  PAST 

PERFORMANCE  WITH  IN  FUTURE  RFPS  AS  AN  EVALUATION  CRTIIERIA 

http:  / /localhost:  8o8o/hii_2  /  publi£h/MMOWGL!_Jdeas/Card_i  &01  .txt 
[20007.00  ~  7.00,  recognize  costjsavings  cost,cdtieria  evaluation,  review  cparE,r^ 
futurejSaving  cost,} 

Card  i^67.txt 

WEIGHT  SOURCE  SELECTION  CRITERIA  SUCH  THAT  ACTUAL  REALIZED  LIFECYCLE 
HAS  HIGHER  RANK  FOR  FUTURE  CONTRACTS  ASYMPTOTICALLY 
http:  / /localhost:  8o8o/bii_2  /  publi5h/M2^IOWGLI_ldea£/ Card_  i467.t!tt 
[20007.00  ~  7.00, weight  source,lifecycle  cost,ltfecyde  realized, savings  oostjfuture 
contracts,selection  source,} 

Card  1484.1x1 

RANK  PAST  PERFORMANCE  ACCORDING  TO  CRITERIA  OTHER  THAN  SUCCESSFUL 
CONTRACT  EXECUTION  SUCH  AS  ACTUAL  FOSS  SOFTTVARE  UCENSES 

http:  / /localhoEt:  8o8o/bii_2/  pubUsh/MMOWGLIJdeas/  Card_i484.tit 
[20007.00  ~  7.00, actual  oostjsavings  costjfoss  software, successful  contract, liceoses 
software,execution  contract,) 

Card  i4Qf:.txt 

REALIZED  PERCENTAGE  OF  EXPENDED  TO  PROGRAMMED  FUNDS  ACTUAL 

LIFECYCLE  O  M  FUNDS  EXPENDED  I 

http:  / /localhost:  8o8o/bii_2/  pubUsh/MMOWGLIJdeas/  Card_14g5.txt 

[20006.00  ~  6.oo,programmed  funds,  realized  CDst,savmgs  cost,lifecyde  actual,expended 

funds,) 

Card  Q6,'^.txt 

APPLY  ADDITIONAL  OBJECTIVES  ON  EACH  CONTRACT  TO  EE  COVERED  IN  THE  EVENT 
OF  ADDITIONAL  TIME 

http:  / /localhost:  8o8o/bii_2/ publish/Ma'IOWGLI_ideas/Card_9  63  .txt 

[20005.00  ~  s.oOjSavings  cost, additional  time,objectives  additional,additionaI  apply,) 


Figure  25.  LLA  Search  Results  for  “Savings  Cost” 
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In  Figure  26,  clicking  the  link  “data  models”  reveals  the  LLA  search  results 
shown  in  Figure  27  and  identifies  the  cards  and  the  OSA  strategy  containing  the 
word  pair  “data  models”.  These  idea  cards  enrich  the  concept  “data  models”  in  the 
OSA  strategy  with  related  concepts  such  as  “develops  subsystems,”  “open  data” 
(card  959),  “achieve  interoperability”  (card  1854),  “interoperable  data  ,”  “monolithic 
data”  (card  1757),  “exist  models,”  and  “data  streams”  (card  1626). 


Lexical  Links  Updated  on  Fri  Ang  30  10:13:57  2013  Using  ’Anomah’  Word  Pairs 


Id 

3  C\  10  WGLIjdeas  (Online) 

m-fOWGLIJdeas 

0  S  AKi  ro  cliure  -  2  0 1 5  r?  due  ?  d.p  df 

Total  Row  LLA  Score 

More  Links 

1 

Card  1S56 

Card  1S56 

87.00:SVSTZ:MATIC  ZFUSZrS7.0m 

87 

Card  1291 

Card  1291 

80 .00  :DZ  FZXSZ  AC  OUI S I T I  OXi  80 .004 

80 

3 

Card  1967 

Card  1967 

80.00:ACOUISITIOX  OSAC80.00) 

80 

4 

Card  1087 

Card  1087 

7  6 .00  :.W..AXIXGFLX  \IE.TR1C  S f 7  6 .004 

76 

5 

Card  1554 

Card  1554 

76.00:NfETRICS  O'SA{76.00) 

76 

L 

Card  2159 

Card  2139 

7 1 .00:PEER  mTEWC7 1 .00) 

71 

7 

Card  2155 

Card  2130 

7 1  .OOiPEER  RE\TEW(7 1 .00) 

71 

S 

fA-d  14C] 

r.ird  )4Q] - 

7]  00-PFFB  BF\7FWY7]  00) - 

U - 

9 

Card  2063 

Card  2063  || 

P6.00;EXERCISE  DATAr66.00) 

66 

[lO 

Card  1757 

Card  1757 

b6.00:MODELS  DATAr66.00) 

66 

n\ 

Card  1626 

Card  1626  ||66.00;MODELS  D.AT.A(66.00) 

66 

12 

Card  1S54 

Card  1S54  ||66.00;MODELS  D.AT.Af 66.001 

66 

13 

Card  959 

Card  959  ijdd.OOiMODELS  D.ATACd6.00) 

66 

14 

Card  1438 

Card  1438  jjed.OOA-LAXAGEMEXT  PROGRA\:C64.00) 

64 

15 

Card  1107 

Card  1107  ll58.00:EXGIXEEIUXG  SYSTEMf58.001 

58 

16 

Card  1065 

Card  1065  il31.00;LIXES  PRODUCTf31.00) 

31 

17 

Card  1060 

Card  1060  pl.OOTKES  PRODUCTrdl.OO) 

31 

18 

Card  1061 

Card  1061  Il3 1. 00 :LD!ES  PRODUCTr3 1.001 

31  1 

Figure  26.  A  Match  Matrix  for  the  biiMMOWGLi  Game  Round  2  Cards 
Matched  With  the  OSA  Strategy  Document  Using  ‘Anomaly’  Word 

Pairs 
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LLA  Search  Results 

HODELS  Dx4TA*  r^raed  5  results 

Card  QfiQ.txt 

YOU  SHOULD  EE  DEVISING  DIRECTING  SPECMC  OPEN  FORSYSTEMS  AND 

THEN  INDUSTRY  DEVELOPS  SUBSYSTEMS  THAT  MEET  THAT  MODEL 

http:  / /localhost:  8o8o/bii_2  /  publtsh/MMOWGLI_ideas/  Card_959  .txt 

C20006.00  ~  6.00, develops  subsystemsjdirecting  specific, models  data^data  opea^develops 

industryj) 

Card  i8f:4.txt 

DEFINE  COMMON  PROTOCOLS  AND  THAT  CAN  ACHIEVE 

INTEROPERABILITY  BETWEEN  HARDWARE  SOFTWARE  APPUCAHONS 
http:  / /ItKalhost:  8o8o/bii_2  /  publLsh/MaiOWGLI_ideas/  Card_i  854.txt 
[2ocx>5.oo  ~  5.<x>j  interoperability  achieve, define  common^models  data^applications 
softwaroj) 

Card  I7^7.txt 

ISAMONOUTHIC  MODEL  MORE  DESIRABLE  THAN  MULTIPLE  INTEROPERAELE 

WITH  ADEDUPOCAHON  RESOLUAHON  PROCESS 
http:  / /localhost:  8o8o/bii_2  /  publis:h/MMOWGLI_ideas/  Card_i7^.txt 
[20005.00  ~  5.oOjinteroperable  data,monolithic  data,modeI  data, models  data^) 

Card  l626.txt 

DIFFERENT  EXIST  FOR  DIFFERENT  STREAMS  REF 

http:  / /localhost:  8o8o/bii_2  /  publLsh/MMOWGLI_ideas/  Card_  1 62  6.txt 
[^0004. 00  ~  4.oOjexi£t  model£,models  data^streams  data^} 

OSAE  rochure-  2  o  ir^reduced.p  df 

INCLUDING  STANDARDIZED  SPECMCAHONS  ARCHITECTURES 
INTEROPERABILITY  PROTOCOI^ 

http:  / /localhost:  8o8o/bii_2/  publish/  OSxA_strategy/ OSx4Erochure-20i  3  reduced.pdf 
[20002.00  ~  2.oo^models  data^) 

Back 


Figure  27.  LLA  Search  Results  for  “Models  Data” 

We  show  here  that  a  match  matrix  from  LLA  sorts  out  the  most  interesting 
idea  cards  that  match  the  business  processes  such  as  the  ones  documented  in  the 
OSA  strategy  document  in  the  biiMMOWGLI  game.  LLA  provides  drill-down  and 
search  capabilities  to  show  how  the  concepts  and  ideas  are  presented  in  the  original 
context  and  how  related  ideas  enrich  the  ones  in  the  links. 

The  linked  and  enriched  concepts  can  be  used  as  the  bases  to  apply  the 
collective  intelligence  generated  from  the  brainstorming  MMOWGLI  game  data  to 
improve  the  existing  business  processes.  For  example,  some  of  these  concepts 
were  included  in  the  action  plans:  incentive  (actions  15  and  16  about  rewards  and 
action  plan  21  about  profitability),  life  cycle  and  cost  savings  (action  28,  action  21 
profitability),  and  OSA  acquisition  (action  29). 
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The  idea  cards  data  also  suggests  that  there  could  be  additional  topics  for  in-depth 
discussions  which  were  not  included  in  the  current  action  plans.  Examples  include 
the  following: 

•  Open  standards  and  data  models; 

•  Meaningful  metrics,  OSA  metrics  and  program  metrics;  and 

•  Consolidated  product  lines  based  on  open  standards,  TRF  level  and 
TRF  attributes 

Conclusions 

We  demonstrated  the  use  of  the  MMOWGLI  social  media  brainstorming 
platform  and  LLA  as  a  combined  collective  intelligence  platform  to  gather 
consensus.  We  identified  new  concepts  reflected  in  the  LLA  word  pairs  that  can  be 
linked  to  critical  variables  and  elements  in  these  business  processes  (bii). 

We  used  match  matrices  for  each  individual  theme  found  through  LLA  to 
identify  word  pairs  and  used  these  word  pairs  to  identify  opportunities  in  the  current 
processes.  For  example,  we  found  that  the  great  majority  of  Navy  programs  are 
affected  by  (or  even  critically  dependent  on)  energy  issues,  but  showed  that  goals 
and  even  terms  are  handled  inconsistently.  Without  imposing  significant  operational 
burdens  and  vulnerabilities,  innovative  “energy  efficiency”  ideas  from  the  social 
media  game  might  be  quickly  and  naturally  implemented  into  the  current  processes 
that  drive  force  structures,  combat  operations,  logistics,  and  acquisition  decisions. 
We  identified  these  gaps  and  opportunities,  which  are  listed  Appendix  A. 

LLA  sorts  and  prioritizes  idea  cards  that  might  be  good  candidates  to  engage 
MMOWGLI  action  plans.  For  example,  in  the  biiMMOWGLI  game,  themes 
discovered  using  LLA  should  be  used  in  future  MMOWGLI  games  to  guide  the 
action  plans.  As  shown  in  Figure  28,  the  themes  are  sorted  according  to  their 
relevance  to  the  OSA  strategy  document:  relevance  defined  as  the  percentage  of 
the  number  of  word  pairs  in  the  OSA  strategy  over  the  total  number  of  word  pairs  ( 
e.g.  12/71=16.9%  in  the  first  row).  The  last  column  in  Figure  28  shows  if  the  current 
action  plans  in  the  bii  game  cover  a  theme.  As  seen,  some  themes  are  covered; 
however,  many  themes  can  be  discussion  topics  for  future  action  plans  or  can  be  the 
basis  of  seed  questions  for  future  games. 

Also  in  Figure  28,  the  themes  with  higher  relevance  to  OSA  strategy  indicate 
consensus  between  the  thoughts  of  the  acquisition  community  and  current  OSA 
strategy.  Conversely,  the  themes  with  lower  relevance  to  OSA  strategy  indicate  gaps 
between  the  thoughts  of  the  acquisition  community  and  current  OSA  strategy.  The 
gap  areas  were  discussed  more  in  the  current  game  than  the  consensus  areas. 
Figure  28  can  be  used  to  improve  the  future  game  or  OSA  strategy. 
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We  demonstrated  that  MMOWGLI  together  with  LLA  can  be  used  as  an 
important  tool  throughout  the  longer  lifecycle  of  the  acquisition  process  to 
incorporate  collective  intelligence  from  the  brainstorming  social  media  such 
energyMMOWGU  and  biiMMOWGLI  games  into  improve  DoD  acquisition 
processes. 


rheme  Id 

All  Sources 

MMOWGLIJdeas 

OSAstrategy 

Theme  Keywords 

Overlap 

Relevance  to  OSA  Strategy 

Relevance  to  Action  Plans 

USIAI 

71 

58 

12 

REVIEW, ACaPROCESSES 

1 

16.9% 

No 

174(A) 

87 

76 

9 

QUALITY.COMPONENT, HIGHER 

2 

10.3% 

No 

t35|P| 

194 

170 

20 

OSA,CHANGE,PERSONNEL 

4 

10.3% 

No 

J(A) 

80 

69 

8 

ACQUISITION, TESTING,FULL 

3 

10.0% 

Yes  (action  29) 

68(E) 

157 

138 

15 

DEVELOPMENT,SOFTWARE,DESIGN 

4 

9.6% 

No 

aHA) 

76 

67 

7 

METRICS,STANDARD,SIMILAR 

2 

9.2% 

No 

SKA) 

66 

57 

6 

DATA,  PROPRIETARY,  MODELS 

3 

9.1% 

No 

78(E) 

111 

100 

10 

ENTERPRISE,MONEY,ENABLE 

1 

9.0% 

No 

103(E) 

175 

160 

15 

MULTIPLE,INCENTIVES,FUNDING 

0 

8.6% 

Yes  (action  15,16  &26) 

t98(P| 

202 

181 

15 

SYSTEM,SYSTEMS,OPEN 

6 

7.4% 

No 

186(E) 

111 

103 

8 

REWARD,LARGE,PROVIDE 

0 

7.2% 

Yes  (action  15,16  &26) 

209(E) 

106 

97 

7 

DESIRED,PROJECT,VENDOR 

2 

6.6% 

No 

|3(Pl 

281 

261 

18 

EXISTING,FUTURE,INNOVATION 

2 

6.4% 

No 

38(E) 

123 

113 

7 

TECHNICAL,FRAMEWORK,FUNDS 

3 

5.7% 

No 

40(E) 

143 

135 

7 

COMMON,INFORMATION,PRIOR 

1 

4.9% 

NO 

104(E) 

133 

127 

6 

INDUSTRY,COMPONENTS,CURRENT 

0 

4.5% 

No 

98(E) 

125 

118 

5 

PERFORMANCE,NAVY,GOVERNMENT 

2 

4.0% 

No 

56(E) 

158 

145 

5 

COST,COSTS,LIFE 

8 

3.2% 

Yes  (action  28) 

50(E) 

130 

126 

4 

CONTRACT,CONTRACTS,RFPS 

0 

3.1% 

Yes  (action  24) 

29(E) 

148 

143 

4 

BUSINESS,MODEL,INCENTIVE 

1 

2.7% 

Yes  (action  26) 

184(E) 

117 

115 

2 

REQUIREMENTS,SOURCE,INTERNAL 

0 

1.7% 

No 

H(A) 

69 

68 

1 

EARLY,  UNITJESTS 

0 

1.4% 

Yes  (action  22) 

102(A) 

92 

91 

1 

TECH,IDEA,PROPOSALS 

0 

1.1% 

No 

I«7(A) 

93 

92 

1 

REWARDS,VICE,MEASURE 

0 

1.1% 

Yes  (action  15,16  &26) 

163(E) 

112 

109 

1 

RIGHTS,LICENSE,PROCESS 

2 

0.9% 

Yes  (action  18) 

189(E) 

181 

180 

1 

IP,RISK,CONTRACTOR 

0 

0.6% 

Yes  (action  18) 

133(A) 

74 

74 

0 

STRUCTURE,ORGANIZATIONAL,SUCCESS 

0 

0.0% 

No 

51(A) 

68 

68 

0 

PEOPLE,SOLUTIONS,CONTRACTING 

0 

0.0% 

Yes  (action  24) 

Figure  28.  Sorted  Themes  as  Candidates  for  Action  Plans 


Recommendations  for  Future  Work 

Crowd  sourcing  can  be  used  to  provide  meaningful  feedback  to  current 
business  processes  in  cross-cutting  themes  such  as  energy  reduction  and  the 
efficiency  of  business  innovation  initiatives  such  as  OSA  strategy.  In  the  future,  we 
plan  to  build  the  MMOWGLI  game  infrastructure  in  tandem  with  the  LLA 
computational  structure  to  reduce  manual  labor  and  maximize  analyst  flexibility.  We 
will  continue  to  work  on  real  datasets  that  spur  meaningful  analysis,  and  produce 
further  data  visualizations  tuned  to  support  focused  analytic  queries  by  players  and 
decision  makers.  For  example,  we  plan  to  optimize  the  following  LLA  and 
MMOWGLI  integration  process  for  a  two-week  future  game: 

•  Step  1 :  Request  the  internal  documents  (e.g.,  PE  documents  or  a  OSA 
strategy  document)  for  a  business  process  prior  to  the  game  for  LLA  in 
order  to  compare  and  generate  match  matrices. 
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•  step  2:  Prepare  the  analysis  from  Monday  to  Wednesday  in  the  first 
week,  and  deliver  the  mid-game  report  including  initial  LLA  themes, 
images,  graphs,  and  visualizations  on  Thursday  night.  Game  Masters 
will  assess  whether  the  mid-game  analysis  appear  helpful  for  the 
second  week  of  the  game.  The  improved  and  accelerated  responses 
appear  to  produce  incremental  products  that  can  accomplish  the 
following: 

o  Help  game  designers,  masters,  and  players  to  view  the  overall 
effectiveness  of  a  game:  for  example,  how  does  a  game 
correlate  with  an  existing  business  process  visually? 
o  Help  game  designers  design  action  plans  from  the  LLA  results 
o  Help  game  players  answer  a  query  or  seed  question  using  the 
drill-down,  search  and  link  capabilities, 
o  Help  game  moderators  notice  areas  of  activity  with  particularly 
high  relevance  using  initial  LLA  graph  images,  LLA  graph 
visualizations  and  analysis  reports. 

•  Step  3:  Generate  the  post-game  report.  We  will  focus  on  how  to  link 
the  collected  MMOWGLI  game  data  to  the  business  processes  for  the 
organizations  involved,  and  build  the  concept  and  framework  of  the 
business  process  via  reinforced  learning. 

We  plan  to  design  and  conduct  a  new  energy  related  MMOWGLI  game  in  a 
two-week  timeframe  and  incorporate  the  LLA  analysis  steps  outlined  previously.  We 
also  plan  to  incorporate  the  most  current  acquisition  artifacts,  for  example,  the 
congressional  budget  process  documents  and  PEs  from 
http://www.dtic.mil/descriptivesum.  We  also  seek  to  measure  the  impacts  of  the 
game  jointly  with  increased  focus  on  key  acquisition  metrics  such  as  cost,  schedule, 
and  performance  to  see  if  the  collective  intelligence  enhanced  through  business 
process  learning  might  be  used  to  improve  the  current  acquisition  process.  With  the 
new  game  data,  there  can  be  new  patterns  of  improvement.  The  improved 
awareness  might  be  brought  into  the  business  process  for  a  significant  and  visible 
improvement.  The  evidence  can  also  be  used  as  the  measurement  of  the  impact  of 
the  MMOWGLI  game  as  our  effort  continues. 

In  addition,  we  see  excellent  potential  in: 

Crowd  sourcing  to  provide  meaningful  feedback  on  either  cross-cutting 
themes  (such  as  energy  reduction/efficiency)  or  specific  acquisition 
programs. 
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Building  the  MMOWGLI  game  infrastructure  in  tandem  with  LLA 
computational  structure  to  reduce  manual  labor  and  maximize  analyst 
flexibility  with  each  round 

Continuing  work  on  real  datasets  that  spurs  meaningful  (rather  than  toy 
or  contrived)  analysis,  and  produce  further  data  visualizations  tuned  to 
support  focused  analytic  queries  by  players  and  decision  makers. 

Maintaining  backwards  compatibility  among  games  to  enable  steady 
growth  via  the  available  corpus  and  products  each  year.  This  further 
enables  longitudinal  analysis  and  observability  of  trends  and  evolution 
over  time. 

Stabilizing  the  data-model  design  of  LLA  computational  products, 
which  may  enable  future  visualization  improvements  to  be  directly 
applied  to  past  products 

Speedier  production  of  LLA  products  which  can  influence  fast-react 
game  rounds  or  program  changes  as  they  proceed,  rather  than  after 
the  event.  We  want  to  reduce  analysis  cycles  from  weeks  to  days,  and 
even  to  hours,  approaching  real  time. 

Program-support  brainstorming  and  collective  intelligence  experiments 
which  should  continue,  both  for  proposed  and  current  programs  of 
record.  Games,  together  with  LLA,  connecting  the  record  of  “what  is 
reported  being  done”  with  “what  do  people  think,”  all  help  normalize  the 
use  of  concept  terminology  and  also  identify  unsuspected  applicability 
of  new  breakthrough  capabilities. 

Overall  progress  and  process  improvements  that  may  now  be 
measured  so  that  causes  and  effects  of  improvements  in  acquisition 
system  cost-effectiveness  and  responsiveness  are  documented. 

Navy  strategies  for  improving  energy  efficiency  that  needs  to  be 
handled  consistently  across  programs.  Terms  of  reference,  metrics, 
opportunities  all  need  to  be  addressed  consciously  and  consistently. 

Following  a  series  of  deliberate  experiments,  long-term  procedural 
improvements  to  the  formal  milestone  acquisition  process  can  be 
considered.  For  example: 

o  Are  program  terms  of  reference  consistent  with  Department¬ 
wide  best  practice? 

o  Are  all  applicable  energy  reduction  and  energy  efficiency 
techniques  identified? 
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o 


Routine  crowd  sourcing  as  due  diligence:  subject-matter  expert 
and  public  reviews  (as  appropriate)  to  accompany  milestone 
decisions 

o  Has  in-game  or  post-game  analysis  identified  synergies  among 
different  programs  that  deserve  further  investigation? 

The  validation  of  LLA  results  have  been  validated  by  domain  experts.  For 
example,  experts  can  visually  examine  the  concepts  extracted  by  LLA  as  shown  in 
Appendix  B. 

In  order  to  achieve  these  long  time  goals,  it  is  important  to  continue  validating 
the  LLA  method  and  integrating  it  with  the  crowd-sourcing  MMOWGLI  platform. 
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Appendix  A.  Gaps  and  Opportunity  Areas  to  integrate 
the  innovative  Concepts  and  Action  Plans  From  the 
MMOWGLI  Energy  Game  Into  Current  Navy  Program 
Elements  (PEs) 


This  appendix  list  the  themes  discovered  by  LLA  and  matches  between 
energyMMOWGU  game  action  plans  and  Navy  PEs.  These  are  the  opportunity 
areas  for  improving  Navy  energy  efficiency. 
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The  match  matrix  for  Theme  430  suggests  that  PEs  mentioned  the  concepts 
“existing  fleet”,  “shipboard  system(s)”,  “shipboard  equipment”  and  “secondary 
power”  that  might  have  the  overall  potential  to  engage  action  plan  10,  26  and  18. 

•  action  plan  10:  In  this  era  of  convergence  reduce  the  number  of 
shipboard  systems  and  focus  more  on  small  computers  with  high 
capability  (Android,  iOS  apps) 

•  action  plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and 
ashore 

•  action  plan  18:  Offshore  basing. 
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The  match  matrix  for  Theme  393  suggests  that  the  PEs  with  the  concepts 
“Navy  energy”,  “energy  systems”,  “power  generation”,  “alternative  fuel”,  “alternative 
energy”,  “renewable  sources”  and  “costs  -  energy/infrastructure”  could  be  used 
good  candidates  to  implement  the  innovative  ideas  related  to  action  plans  11,  18,  22 
and  35. 

•  action  plan  1 1 :  Enhanced  Education  to  Develop  an  Energy  Efficient 
Fleet; 

•  action  plan  18:  Offshore  basing 

•  action  plan  22:  Scaling  the  Small  Solutions:  Energy  Recycling  and 
Rethinking  “The  Big  Fix.” 
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The  match  matrix  for  Theme  458  shows  that  the  PEs  mentioned  “Naval 
expeditionary”,  “ship  board”  and  “strike  carrier,”  which  can  be  good  candidates  to 
engage  action  plan  15  and  26. 
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•  action  15:  A  global  navy  formed  by  an  alliance  of  nation  linked  in  real 
time.  That  way  the  nearest  force  will  respond  and  reduce  travel 
distances. 

•  action  26:  Expand  the  use  of  nuclear  power  in  the  fleet. 

•  Related  concepts  include  “multiple  hardware,”  “operating  time,”  and 
“dashboard  energy” 


The  matrix  for  Theme  905  showed  that  the  PEs  involved  “unmanned 
systems,”  “surface  ships,”  “nuclear  powered,”,  “operational  environment,”  “water 
treatment,”  which  can  be  good  candidates  for  engaging  action  plan  18,  19,  20,26, 
31,35,4  and  7. 

•  action  plan  18:  Offshore  basing 

•  action  plan  19:  Implement  a  self-sustaining  support  infrastructure  on  all 
Navy  bases 

•  action  plan  20:  Sails  on  vessels,  use  sails  that  are  foldable  on  the 
sides  of  vessels. 

•  action  plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and 
ashore 

•  action  plan  31 :  Add  “reducing  energy  consumption”  to  Battle  E  criteria 
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•  action  plan  35:  Create  3D/verticle  farms  for  use  in  growing  biofuels, 
and  crop  for  human  consumption. 

•  action  plan  4:  Change  small  land  vehicle  transportation  to  hybrid 
vehicles  in  non-combat  capacities. 

•  action  plan  7:  Install  “sea  brakes”  that  generate  electricity,  like  a  Prius. 
These  could  be  used  to  aid  in  docking/slowing  ships,  and  reduce  need 
for  tugs. 
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The  match  matrix  for  Theme  132  shows  that  the  PEs  mentioned  “additional 
energy,”  “ground  forces”  (e.g.,  PE  06021 31 M,  PE  0603640M,  PE  020631 3M,  PE 
0602750N,  PE  060501 3M,PE  0604404N),  “harvesting  energy”  (e.g.,  PE  0602236N: 
Warfighter  Sustainment  Applied  Res;  PE  0603673N: 

(U)Future  Naval  Capabilities  Advanced  Tech  Dev;  PE  0601 153N: 

Defense  Research  Sciences;  PE  06021 23N:  Force  Protection  Applied  Res), 
“potential  energy,”  and  “hydrodynamic  forces,”  which  are  good  candidates  to  engage 
action  plan  14,15,17,18,34  and  7 
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•  action  plan  14:  Recycle  everything  biological  into  fuel. 

•  action  plan  15:  A  global  navy  formed  by  an  alliance  of  nation  linked  in 
real  time.  That  way  the  nearest  force  will  response  and  reduce  travel 
distances. 

•  action  plan  17:  Energy  harvesting  satellites  in  outer  space  transmit 
energy  to  earth  via  microwave  or  laser  beam. 

•  action  plan  18:  Create  flotillas  of  ships  and  sea  platforms  as  off  shore 
bases  in  critical  regions  such  as  the  South  China  Sea. 

•  action  plan  34:  Create  an  online  system  or  suggestion  card  system  for 
Navy  personnel  to  input  where  they  see  energy  savings  in  their  job. 

•  action  plan  7:  Install  "sea  brakes,"  that  generate  electricity,  like  a  Prius. 
These  could  be  used  to  aid  in  docking/slowing  ships  and  reduce  the 
need  for  tugs. 


The  match  matrix  for  Theme  787  suggests  that  “energy  efficiency”  and  “fuel 
efficiency,”  which  can  be  viewed  as  “survivability  requirements,”  therefore,  any  PEs 
related  to  “survivability  requirements”  (e.g.  PE  060321 6N:  Aviation  Survivability)  or 
“operational  requirements”  can  be  used  to  engage  action  plans  10,  11, 20,  27,  31, 
34  and  9. 

•  action  plan  9:  Composite  Ship  Design:  Explore  the  Use  of  Polymer 
Substrates  for  Improved  Ship  Structural  Design 

•  action  plan  10:  In  this  era  of  convergence  reduce  the  number  of 
shipboard  systems  and  focus  more  on  small  computers  with  high 
capability  (Android,  iOS  apps) 
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The  match  matrix  for  Theme  494_shows  that  the  PEs  mentioned  “shared 
information,”  “signal  intelligence,”  “share  data,”  “data  structures,”  “intelligence 
systems,”  “artificial  Intelligence,”  and  “maritime  warfare”  might  be  good  candidates  to 
engage  action  plans  16,  18,  26,  31,  and  36. 

•  action  plan  16:  Use  synthetic  lubricants  to  save  5  -  25%  of  energy 
costs 

•  action  plan  18:  Create  flotillas  of  ships  and  sea  platforms  as  off  shore 
bases  in  critical  regions  such  as  the  South  China  Sea 

•  Action  plan  36:  Become  more  efficient  at  structured,  logical  dialogue  to 
find  the  solutions  you  seek 
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The  match  matrix  for  Theme  633_suggests  that  the  PEs  mentioned  “advanced 
tech”  (e.g.  PE  0603673N:  (U)Future  Naval  Capabilities  Advanced  Tech  Dev), 
“greater  efficiency”  (e.g.  PE  0603747N:  Undersea  Warfare  Advanced  Tech)  and 
“power  plants,”  which  can  be  good  candidates  to  engage  action  plansi  1,21,  and  4. 

•  action  plan  1 1 :  Enhanced  Education  to  Develop  an  Energy  Efficient 
Fleet 

•  action  plan  21:  DOD  Shore  Facility  Energy  Independence:  Explore  use 
of  Thorium-Based  Reactors  (LFTR-Liquid  Flouride  Thorium  Reactor) 
for  power  generation  off  the  grid. 

•  action  plan  4:  Change  small  land  vehicle  transportation  to  hybrid 
vehicles  in  non-combat  capacities. 
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The  match  matrix  for  Theme  326  suggests  that  the  PEs  mentioned  “energy 
security,”  “missile  defense,”  “operational  security,”  “cyber  security,”  “national 
security,”  and  “naval  postgraduate  school,”  which  might  be  good  candidates  to 
engage  action  plans  17,  19,  4,  27,  4,  35,  and  5. 

•  action  plan  17:  Energy  harvesting  satellites  /  Space  based  solar  power. 

•  action  plan  19:  Implement  self-sustaining  support  infrastructure  on  all 
Navy  bases. 

•  action  plan  4:  Change  small  land  vehicle  transportation  to  hybrid 
vehicles  in  non-combat  capacity. 


id 

na\7 2C13(On  ne 

actions 16 0.53.txt 

act  on5 18 C.7i.txt 

actions 21 0.67.txt 

act  on5 25 C. 88.txt 

actions  26  i.44.txt 

actlons 31 1.10.txt 

acTon5 34 l.CC.txt 

act  on5 9 C.65.t3(t 

Total  Row  LLA Score 

1 

-3.cd 

NIJ CLEAR  ^ljvvER 

NUCLEAR  POWER 

NUCLEAR  FLEET;NUCLEAR  POWER;NUCLEAR  NAVAL 

3615 

2 

:6::35':n  -  pb 

- cl 

NUCLEAR  COWER 

NUCLEAR  POWER 

NUCLEAR  POWER;NUCLEAR  TECHNOLOGY 

2892 

3 

-3  cd 

NUCLEAC  cnvvER 

NUCLEAR  POWER 

NUCLEAR  POWER 

2169 

4 

'-3  i;c 

L03  S-  C5  S--'S-EV15 

E-AMjARjS  CCMMiCN 

L  '  1-li-^jE  lEir 

2169 

5 

Z6Z5Z:iU  5  p=! 

-3,i;d 

LOG  S'  C 5  S-S'EVIS 

L  j  1-[J-jE  lEN 

1446 

6 

cp::;:5?n  -  pe; 

33. cd 

L  j  '  !-[J-  E  lEtr 

WAN  OAR  05  OEvELCWMEIW 

1446 

7 

ChP-lE.N  5  P3 

-3,cd 

LOG  S'  CS  MANAGEMEN" 

WAN  OAR  OS  OA'A 

1446 

8 

=  -  - 

13. cd 

L  j  '  1-IJ-  E  lEir 

723 

9 

:6::-:.5N  5  P3 

-3  -d 

WAN  OAR  OS  OEvELO^MELW 

723 

10 

ip:-:-',  e  ^e 

-d 

L  J  1-IJ-jE  lEir 

723 

11 

CfcPSEESN  -  P3 

.3  d 

WAN  OAR  05  OEvELCWvlEI-r 

723 

12 

:6:36-:\i  3  -3 

pd 

723 

13 

C6::356:N  -  P3 

-  5 . 1C  d 

NUCLEA=''ECHM0LCG'-' 

723 

14 

CfiCSlE.SN  3  PB 

:3Dd 

5-ANDARD5  SAFETY 

723 

The  match  matrix  for  Theme  917_suggests  that  the  PEs  mentioned  “nuclear 
power,”  “nuclear  technology,”  “safety  standards,”  “logistics  systems,”  “logistics 
management,”  “standards  development/data,”  and  “common  standards,”  which 
might  be  good  candidates  to  engage  action  plans  16,  18,  25,  26,  31,34  and  9. 
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•  action  plan  34:  Create  an  online  system  or  suggestion  card  system  for 
Navy  personnel  to  input  where  they  see  energy  savings  in  their  job 
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The  match  matrix  for  Theme  579  suggests  that  the  PEs  mentioned  “energy 
management,”  “composite  materials,”  “processing  capabilities,”  “supply  chains,” 
“electrical  energy,”  “hazardous  waste,”  “energy  absorbing,”  “sinks  heat,”  “heat 
reduce,”  and  “naval  academy,”  which  might  be  good  candidates  to  engage  action 
plans  8,  20,  26,  and  9. 

•  action  plan  8:  Shore  Energy  Optimization  Strategy-Recommendations 
for  Improvements  and  Implementation. 


The  match  matrix  for  Theme  854_suggests  that  PEs  mentioned  “turbine 
engine,”  “diesel  engine,”  “energy  sources,”  “power  sources,”  and  “greenhouse  gas,” 
which  might  be  good  candidates  to  engage  “behavior  modification”  related  to  action 
plans  27,  8,  and  5. 

•  action  plan  27:  Upgrade  Navy  housing  with  SMART  Grids  to  reduce 
energy  consumption.  By  individualizing  electricity/utility  bills  to  single 
households,  family  users  will  be  motivated  to  increase  energy  saving 
efforts 

•  action  plan  5:  Incentivize  behavior  to  reduce  electricity  usage  in  Navy 
housing 

•  action  plan  8:  Update  older  buildings  to  be  more  energy  efficient.  The 
Navy  is  still  using  buildings  that  are  almost  a  century  old. 

These  PEs  include,  for  example,  PE  0603573N:  Advanced  Surface  Machinery  Sys, 
PE  0603724N:  Navy  Energy  Program,  PE  0205633N:  Aviation  Improvements,  PE 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  54  - 

NAVAL  POSTGRADUATE  SCHOOL 


0206623M:  MC  Ground  Cmbt  Spt  Arms  Sys,  and  PE  0605864N:  Test  &  Evaluation 
Support. 
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They  might  be  good  candidates  to  engage  action  plans  that  mention  “mobile 
power,”  “electric  warship,”  “training  centers,”  and  “ocean  wave.”  These  action  plans 
include 

The  match  matrix  for  Theme  732  suggests  that  the  PEs  mentioned  “ship 
surface,”  “fleet  surface,”  “power  management,”  “ship  power,”  “supplying  power,”  and 
“generating  power.”  These  PEs  include,  for  example, 

•  PE  0603563N:  Ship  Concept  Advanced  Design 

•  PE  06021 23N:  Force  Protection  Applied  Res 

•  PE  0603573N:  Advanced  Surface  Machinery  Sys 

•  PE  0206624M:  Marine  Corps  Cmbt  Services  Supt 

PE  06031 14N:  Power  Projection  Advanced  Technology 

•  PE  0601 153N:  Defense  Research  Sciences 

•  PE  06021 31 M:  Marine  Corps  Lndg  Force  Tech 

They  might  be  good  candidates  to  engage  action  plans  that  mention  “mobile  power,” 
“electric  warship,”  “training  centers,”  and  “ocean  wave.”  These  action  plans  include 
action  plans  23  and  1 1 . 
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‘action  plan  23:  Combine  Global  Homeporting  with  Localized  Energy 
Generation  Across  the  Globe. 

‘action  plan  1 1 :  Enhanced  Education  to  Develop  an  Energy  Efficient  Fleet 
and  engage  major  universities  to  create  a  cross  disciplinary  curriculum  for  “energy 
design”  in  all  fields  for  all  forms  of  energy. 


The  match  matrix  for  Theme  449  suggests  that  the  PEs  mentioned  “power 
projection,”  which  can  be  used  to  engage  “social  media”  for  “fuel/energy  saving.” 

•  Action  1 1 :  Enhanced  Education  to  Develop  an  Energy  Efficient  Fleet 
and  engage  major  universities  to  create  a  cross  disciplinary  curriculum 
for  “energy  design”  in  all  fields  for  all  forms  of  energy. 
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.'FE-'-  .jfJi-  =LEE 

1298 

IS 

:3:53'31.1  3  FE  2213  cd' 

CFE-'-  ctj:  -tee--':- 

1298 

19 

2325:5-11  3  FE  2213  cd' 

2FE  =  -  2112  =LEE 

1298 

20 

232323311  3  FE  2213  i:d^ 

2FE=---  2112  ='E2E-  =  2- 

1298 

21 

232373  =  11  -  FB  2213. i;d' 

;2112-='LI:2-  211  1.1  L  -in--' 

1298 

22 

22:532:11  7  FE  2213  cd^ 

22112  =U2-  211  l.I  L  --=  ■ 

1298 

73 

2322-3511  2  FE  22:3  cd^ 

2FE='-  2112  =LEE- 

1298 

7- 

232223311  2  FE  2213  i;d^ 

2FE  =  C-  2112  ='E2E-='2- 

1298 

75 

:3:53::ii  7  fe  22:3  i:d^ 

2FE  =  --  2112  =E2E-=2- 

1298 

The  match  matrix  for  Theme  682  suggests  that  the  PEs  mentioned  “ship 
construction,”  “ship  operations,”  “fleet  operations,”  “military  construction,”  “operations 
research,”  which  can  be  good  candidates  to  engage  action  plans  10,  26  and  6. 

•  action  plan  10:  In  this  era  of  convergence  reduce  the  number  of 
shipboard  systems  and  focus  more  on  small  computers  with  high 
capability  (Android,  iOS  apps) 

•  action  plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and 
ashore 
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•  action  plan  6:  Implement  large  umbrellas  for  ships  to  use  shading  to 
keep  ships  cooler  and  also  use  "carport"  structures  for  ships  docked  on 
the  pier 


Id 

navv 2013(Online) 

actions 16 0.53.txt 

acti  ons 18 0.7 1  .txt 

actions 27 0.88.txt 

actions 28 0.86.txt 

actions 34 1.00.txt 

actions 35 0.82.txt 

Total  Row  LLA  Score 

2 

0205633N  7  PB  2015  iicl- 

PAR”  LIFE 

SPARE  PARTS 

2130 

3 

0205604N  7  PB  zCli  ircP' 

COMMUNICATION  DATA 

1065 

4 

06042 SON  5  PB  2C:5  i:d-' 

PROGRAMMABLE  RADIO 

1065 

5 

0604307 N  5  PB  2C:;  ir  d' 

PARTS  REPLACEMENT 

1065 

6 

0206624M  7  PB  2C:;  i:.d-’ 

COMMUNICATION  EQUIPMENT 

1065 

7 

0605853N  6  PB  2C::.  i:d-' 

GUIDANCE '2U:  -  0-:T  NG 

1065 

8 

0603542N  4  PB  2215  i;d- 

PARTS  REPLACEMENT 

1065 

9 

0206313M  7  PB  22:5  iid- 

Cl"  MM  UN  CA^  ON  OAtA 

1065 

10 

0602 7 SON  2  PB  22:2  i:d^ 

URBAN  ENV  RON  MEN'S 

1065 

11 

0604503 N  5  PB  22:2  ird-’ 

COMMUNICATION  EQUIPMENT 

1065 

12 

0604404N  5  PB  22  2  2  ir  d-' 

:V  NG  A  R 

1065 

13 

0603271N  3  PB  2222  i::d^ 

PARTS  REPLACEMENT 

1065 

14 

0604231N  5  PB  2222  i:d- 

COMMUN  CA"  ON  JA-A, 

1065 

The  match  matrix  for  Theme  257  suggests  that  the  PEs  mentioned  “parts 
replacement,”  “communication  equipment,”,  “air  wing,”  “communication  data,”  and 
“urban  environments,”  which  might  be  good  candidates  for  action  plans  16,  18, 
27,28,  34  and  35 

•  action  plan  16:  Use  synthetic  lubricants  to  save  5-25%  of  energy 
costs. 

•  action  plan  18:  Offshore  basing. 

•  action  plan  27:  Upgrade  Navy  housing  with  SMART  Grids  to  reduce 
energy  consumption.  By  individualizing  electricity/utility  bills  to  single 
households,  family  users  will  be  motivated  to  increase  energy  saving 
efforts. 

•  action  plan  28:  Power  on-board  minor  electronics  with  stationary  bikes 
used  for  personnel  fitness  training 

•  action  plan  34:  Online  Feedback  &  Social  Networking 

•  action  plan  35:  3D  farming-Less  land  use  and  local  agriculture 
reducing  fuel  use  and  potential  location  of  bio-fuel  crops. 


The  match  matrix  for  Theme  198  suggests  that  the  PEs  mentioned  “energy 
saving,”  “fuel  savings,”  “cost  savings,”  “fuel  cell,”  “cell  technologies,”  “storage 
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energy,”  and  “storage  systems,”  which  might  be  good  candidates  to  engage  action 
plans  related  to  these  concepts. 

The  matrices  that  resulted  from  this  task  will  help  design  the  specific 
questions  to  address  the  issues  in  a  program-to-program  basis  to  continue  the 
energyMMOWGU  game  with  acquisition  professionals  on  the  acquisition  research 
community  in  the  future. 
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Appendix  B.  Visualizations  for  Themes  Identified  in 
biiMMOWGL!  Game  Round  2 


This  appendix  lists  sample  themes  in  Figure  19.  The  red  links  represent  the 
word  pairs  or  concepts  shared  by  the  idea  cards  and  the  strategy  book.  The  green 
links  represent  the  word  pairs  unique  to  the  strategy  book.  The  yellow  links 
represent  the  word  pairs  or  concepts  unique  to  the  idea  cards.  Each  theme  is 
labeled  using  the  words  in  the  red  nodes.  Word  pairs  shared  in  both  idea  cards  and 
the  strategy  are  red  links.  Word  pairs  unique  to  the  strategy  book  that  are  not 
discussed  in  the  biiMMOWGLI  game  Round  2  are  green  links.  Word  pairs  unique  to 
the  idea  cards  which  are  not  mentioned  in  the  current  strategy  and  considered  as 
interesting  and  crowd-sourced  ideas  are  yellow  links. 


Figure  B1.  Theme  Centered  Around  “Open,  System,  Systems” 

In  Figure  B1  word  pairs  shared  in  both  idea  cards  and  the  strategy  (red  links) 
include  “open  systems,”  “open  standards,”  “enterprise  architecture,”  and  “insert 
capability.”  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were  not 
discussed  in  the  biiMMOWGLI  game  Round  2  include  “OOC  systems,”  “TRFS 
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system,”  “constructed  system,”  “relative  standards,”  “  Naval  open,”  “accommodate 
open,”  “architecture  strategy.”  Word  pairs  unique  to  the  idea  cards  (yellow  links) 
which  are  not  mentioned  in  the  current  strategy  and  considered  as  interesting  and 
crowd-sourced  ideas  include  “combat  system,”  “weapon(s)  system,”  “accounting 
system,”  “systems  availability,”  “legacy  system,”  “technology  insertion,”  “COTS 
insertion,”  etc. 


Figure  B2.  Theme  Centered  Around  “Personnel,  OSA,  Change” 


In  Figure  B2,  word  pairs  shared  in  both  idea  cards  and  the  strategy  (red  links) 
include  “OSA  strategy,”  “assess  OSA,”  “OSA  progress.”  Word  pairs  unique  to  the 
strategy  book  (green  links)  that  were  not  discussed  in  the  biiMMOWGU  game 
Round  2  include  “Timely  OSA,”  “timely  target,”  “sponsors  resource,”  “platform  types,” 
“strategy  execution,”  “guidebook  execution,”  “acquisition  change,”  “successful 
personnel.”  Word  pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not 
mentioned  in  the  current  strategy  and  considered  as  interesting  and  crowd-sourced 
ideas  include  “OSA  infrastructure,”  “OSA  proof,”  “OSA  platform,”  “disruptive  change,” 
“personnel  burnout,”  “personnel  requirements,”  and  “personnel  effectiveness,”  etc. 
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Figure  B3.  Theme  Centered  Around  “Multiple  Funding,  Incentives” 


In  Figure  B3,  there  are  no  word  pairs  shared  in  both  idea  cards  and  the 
strategy  (red  links).  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were 
not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “adjust  funding,”  “adjusting 
incentives,”  “integrated  platform,”  “multiple  platforms,”  “highly  integrated.”  Word  pairs 
unique  to  the  idea  cards  (yellow  links)  which  are  not  mentioned  in  the  current 
strategy  and  considered  as  interesting  and  crowd-sourced  ideas  include  “ECP 
incentives,”  “  industry  incentives,”  “discuss  incentives,”  “motivate  incentives,” 
“contract  incentives,”  “profit  incentives,”  “incentives  term,”  “positive/negative 
incentives,”  etc. 
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Red  nodes;  the  most  "central"  nodes,  used  as  keywords  to  summarize  this  theme 
Red  Links:  shared  by  at  least  two  sources 
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(ii) 

Figure  B4.  Theme  Centered  Around  “Life,  Cost,  Costs” 

In  Figure  B4(i),  word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “total  ownership”,  “ownership  cost(s),”  “life  cycle  cost,”  “system  life,” 
“program  life,”  “cost  savings.”  Word  pairs  unique  to  the  strategy  book  (green  links) 
that  were  not  discussed  in  the  biiMMOWGU  game  Round  2  include  “reduced  cycle,” 
“reduced  costs,”  “reduced  total.”  Word  pairs  unique  to  the  idea  cards  (yellow  links) 
which  are  not  mentioned  in  the  current  strategy  and  considered  as  interesting  and 
crowd-sourced  ideas  include  “cost  tracking,  “indirect  cost,”  “cost  infrastructure,” 
“infrastructure  requirements,”  “realized  lifecycle,”,  “actual  lifecycle,”  “lifecycle 
product,”  “evaluate  compliance,”  “evaluate  metrics,”  “contract  cycle,”  “IP  ownership,” 
etc.  When  highlighting  these  word  pairs,  we  used  Figure  B4(ii)  where  LLA  detected 
more  important  keywords  in  the  inner  circle  and  more  popular  keywords  in  the  outer 
ring. 
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(ii) 

Figure  B5.  Theme  Centered  Around  “Software  Development,  Design” 

In  Figure  B5(i),  word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “software  development,”  “development  environments,”  “Naval 
acquisition.”  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were  not 
discussed  in  the  biiMMOWGU  game  Round  2  include  “development  concept,” 

“Naval  enterprise,”  “Naval  OSA,”  “technical  practices,”  “communication  materials,” 
“modular  driven,”  “training  guide,”  “iterative  set.”  Word  pairs  unique  to  the  idea  cards 
(yellow  links)  which  are  not  mentioned  in  the  current  strategy  and  considered  as 
interesting  and  crowd-sourced  ideas  include  “iterative  development,  “iterative 
design,”  “agile  model,”  “agile  development,”  “agile  processes,”  “incremental 
development,”  “incremental  funding,”  “reuse  driven,”  “costs  driven,”  “open 
communication,”  “contractor  communication,”  “existing  practices,”  “practices 
sustainment.”  etc.  When  highlighting  these  word  pairs,  we  used  Figure  B5(ii)  where 
LLA  detected  relatively  important  keywords  in  the  inner  circle  and  popular  keywords 
in  the  outer  ring. 
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(ii) 

Figure  B6.  Theme  Centered  Around  “Business  Modei,  Incentive” 

In  Figure  B6(i),  word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “business  model.”  Word  pairs  unique  to  the  strategy  book  (green  links) 
that  were  not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “integrates 
business,”  “integrates  approach,”  “acquisition  model.”  Word  pairs  unique  to  the  idea 
cards  (yellow  links)  which  are  not  mentioned  in  the  current  strategy  and  considered 
as  interesting  and  crowd-sourced  ideas  include  “big  data,”  “big  program,”  “big 
Navy,”  “big  companies,”  “big  gov,”  “big  savings,”  “big  money,”  “integrates  apps,” 
“integrate  android,”  etc.  When  highlighting  these  word  pairs,  we  used  Figure  B6(ii) 
where  LLA  detected  relatively  important  keywords  in  the  inner  circle  and  popular 
keywords  in  the  outer  ring. 
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Figure  B7.  Theme  Centered  Around  “Common,  Prior,  Information” 


In  Figure  B7,  word  pairs  shared  in  both  idea  cards  and  the  strategy  (red  links) 
include  “common  OSA.”  Word  pairs  unique  to  the  strategy  book  (green  links)  that 
were  not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “increase 
interoperability,”  “interoperability  protocols,”  “common  TRFS.”  Word  pairs  unique  to 
the  idea  cards  (yellow  links)  which  are  not  mentioned  in  the  current  strategy  and 
considered  as  interesting  and  crowd-sourced  ideas  include  “common  data,” 
“common  supply,”  “common  software,”  “common  RFP,”  “common  source,”  “common 
test(ing),”  “common  requirements,”  “common  protocols,”  “legacy  interoperability,” 
etc. 
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Figure  B8.  Theme  Centered  Around  “Current,  Industry,  Component” 

In  Figure  B8,  there  are  no  word  pairs  shared  in  both  idea  cards  and  the 
strategy  (red  links).  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were 
not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “current  programs,” 
“reusable  components,”  “enterprise  adoption,”  “program  adoption.”  Word  pairs 
unique  to  the  idea  cards  (yellow  links)  which  are  not  mentioned  in  the  current 
strategy  and  considered  as  interesting  and  crowd-sourced  ideas  include  “current 
contracting,”  “current  data,”  “current  profit,”  “current  RFP,”  “current  acquisition,” 
“current  proprietary,”  “industry  vendors,”  “rewarding  industry,”  “industry 
understanding,”  “encourages  industry,”  “encourages  IP,”  “encourages  FOSS,”  etc. 
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Figure  B9.  Theme  Centered  Around  “RFPs,  Contract,  Contracts” 


In  Figure  B9,  there  are  no  word  pairs  shared  in  both  idea  cards  and  the 
strategy  (red  links).  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were 
not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “contract  solicitations,” 
“contract  language,”  “language  templates,”  “create  contract.”  Word  pairs  unique  to 
the  idea  cards  (yellow  links)  which  are  not  mentioned  in  the  current  strategy  and 
considered  as  interesting  and  crowd-sourced  ideas  include  “SI  contract,”  “IDIQ 
contract(s),”  “contract  style,”  “RFPS  pillars,”  etc. 
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Figure  B10.  Theme  Centered  Around  “Government,  Navy  Performance” 


In  Figure  B10,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “program  executive.”  Word  pairs  unique  to  the  strategy  book  (green 
links)  that  were  not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “involved 
workforce.”  “executive  offices”.”  Word  pairs  unique  to  the  idea  cards  (yellow  links) 
which  are  not  mentioned  in  the  current  strategy  and  considered  as  interesting  and 
crowd-sourced  ideas  include  “Naval  leadership,”  “leadership  performance,” 
“government  stakeholders,”  “government  procurement,”  “initial  procurement,”  “risk 
involved,”  “IP  involved,”  “stakeholder  involved,”  “employees  involved,”  “drive 
software,”  “drive  design,”  “drive  innovation,”  etc. 
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Figure  B11.  Theme  Centered  Around  “Technical,  Framework,  Funds” 


In  Figure  B1 1 ,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “consolidate  technical,”  “technical  frameworks.”  Word  pairs  unique  to 
the  strategy  book  (green  links)  that  were  not  discussed  in  the  biiMMOWGLI  game 
Round  2  include  “technical  designs,”  “technical  OSA,”  “technical  reference,” 
“reference  framework(s),”  “framework  elements,”  “TRF  elements,”  “volatile 
elements.”  Word  pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not 
mentioned  in  the  current  strategy  and  considered  as  interesting  and  crowd-sourced 
ideas  include  “proprietary  framework,  “testing  framework,”  “consolidated  framework,” 
“EA  framework,”  “framework  support,”  “OSA  framework,”  “framework  architecture,” 
“acquisition  elements,”  “reference  implementation,”  “open  framework(s),”  etc. 
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In  Figure  B12,  there  are  no  word  pairs  shared  in  both  idea  cards  and  the 
strategy  (red  links).  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were 
not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “user  requirements.”  Word 
pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not  mentioned  in  the  current 
strategy  and  considered  as  interesting  and  crowd-sourced  ideas  include  “OSA 
requirements,”  “data  requirements,”  “nonconforming  requirements,”  “performance 
requirements,”  “customer  requirements,”  “system  requirements,”  “requirements 
oriented,”  “internal  costs,”  “internal  innovation,”  “internal  profit,”  “internal  investment,” 
“internal  funds,”  “internal  spend,”  “internal  rate,”  “performance  evaluation,” 
“evaluation  team,”  “evaluation  metrics,”  “evaluation  driven,”  “success  rate,”  “rate 
contributions,”  “rate  changing,”  “user  community,”  “test(ing)  community,”  “license 
shift,”  “funding  shift,”  “lA  requirements,”  “lA  certification,”  “centralize  lA,”  “reduce  lA,” 
“certification  process,”  “CISSP  certification,”  “certification  phase,”  “operational 
scenarios,”  “operational  capability,”  “manpower  capability,”  “upgrade  impact,”  etc. 
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Figure  B13.  Theme  Centered  around  “License  Rights,  Process” 


In  Figure  B13,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “data  rights,”  “restricted  rights.”  There  are  no  word  pairs  unique  to  the 
strategy  book  (green  links)  that  were  not  discussed  in  the  biiMMOWGU  game 
Round  2.  Word  pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not  mentioned 
in  the  current  strategy  and  considered  as  interesting  and  crowd-sourced  ideas 
include  “(un)limited  rights,”  “digital  rights,”  “modified  rights,”  “squabble  rights,”  “IP 
rights,”  “license  rights,”  “requested  rights,”  “purchasing  rights,”  “IP  license,”  “OSA 
license,”  “license  model,”  “negotiate(d)  license,”  “government  license(s),”  “license 
renewal,”  “open  licenses,”  “commercial  licenses,”  “software  licenses,”  “common 
licenses,”  “standard  licenses,”  “purchasing  process,”  “prototyping  process,”  “broken 
process,”  “traumatic  process,”  “audit  process,”  “appropriation  process,”  etc. 
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In  Figure  B14,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  “enterprise  architecture.”  Word  pairs  unique  to  the  strategy  book 
(green  links)  that  were  not  discussed  in  the  biiMMOWGLI  game  Round  2  include 
“OSA  enterprise,”  “proven  enterprise,”  “operable  enterprise,”  “enable  enterprise.” 
Word  pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not  mentioned  in  the 
current  strategy  and  considered  as  interesting  and  crowd-sourced  ideas  include 
“enterprise  money,”  “enterprise  COTS,”  “enterprise  commonality,”  “enable 
interfaces,”  “enable  PMS,”  “enable  competition,”  “enable  contractors,”  “sharing 
expertise,”  etc. 
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Figure  B15.  Theme  Centered  around  “Provide,  Large,  Reward” 


In  Figure  B15,  there  are  no  word  pairs  shared  in  both  idea  cards  and  the 
strategy  (red  links).  Word  pairs  unique  to  the  strategy  book  (green  links)  that  were 
not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “reward  mechanisms,” 
“delivery  mechanisms,”  “mechanisms  transparency,”  “provide  transparency,” 
“establish  reward.”  Word  pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not 
mentioned  in  the  current  strategy  and  considered  as  interesting  and  crowd-sourced 
ideas  include  “compliance  mechanisms,”  “FOSS  criteria,”  “FOSS  license,”  “FOSS 
proponent,”  “FOSS  software,”  “biggest  FOSS,”  etc. 
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In  Figure  B16,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  "meaning  metrics,”  “OSA  metrics.”  Word  pairs  unique  to  the  strategy 
book  (green  links)  that  were  not  discussed  in  the  biiMMOWGLI  game  Round  2 
include  “program  metrics,”  “update  metrics,”  “enterprise  metrics,”  “qualification 
standard.”  Word  pairs  unique  to  the  idea  cards  (yellow  links)  which  are  not 
mentioned  in  the  current  strategy  and  considered  as  interesting  and  crowd-sourced 
ideas  include  “common  metrics,”  “metrics  collection,”  “savings  metrics,” 
“performance  metrics,”  “measurable  metrics,”  “financial  metrics,”  “ISO  metrics,”  “ISO 
standard,”  “missile  standard,”  “source  standard,”  “instrumentation  standard,” 
“industry  standard,”  “license  standard,”  “middleware  standard,”  “standard 
categories,”  “IP  categories,”  “acquisition  categories,”  “similar  metrics,”  “similar 
framework,”  “similar  programs,”  “similar  systems,”  etc. 
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In  Figure  B17,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  "peer  review.”  Word  pairs  unique  to  the  strategy  book  (green  links)  that 
were  not  discussed  in  the  biiMMOWGLI  game  Round  2  include  “  review  gate,” 
“technical  review,”  “review  process(es),”  “alignment  processes,”  “acquisition 
processes,”  “communications  processes,”.”  Word  pairs  unique  to  the  idea  cards 
(yellow  links)  which  are  not  mentioned  in  the  current  strategy  and  considered  as 
interesting  and  crowd-sourced  ideas  include  “review  reasons,”  “multiple  reasons,” 
“security  reasons,”  “review  requirements,”  “CPARS  review,”  “sponsor  review,” 
“sponsor  innovation,”  “expedite  sponsor,”  “sponsor  success,”  “ACQ  strat,”  “ACQ 
DAU,”  “professional  ACQ,”  “peer  ACQ,”  etc. 
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In  Figure  B18,  Word  pairs  shared  in  both  idea  cards  and  the  strategy  (red 
links)  include  "data  models,”  “exercise  data  rights.”  Word  pairs  unique  to  the  strategy 
book  (green  links)  that  were  not  discussed  in  the  biiMMOWGLI  game  Round  2 
include  “monolithic  business,”  “decompose  monolithic,”  “monolithic  acquisitions,” 
“modular  acquisitions,”  “OSA  acquisitions.”  Word  pairs  unique  to  the  idea  cards 
(yellow  links)  which  are  not  mentioned  in  the  current  strategy  and  considered  as 
interesting  and  crowd-sourced  ideas  include  “monolithic  contracts,”  “monolithic 
data,”  “accessible  data,”  “proprietary  data,”  “data  learned,”  “open  data,”  “data 
mining,”  “data  analytics,”  data  flows,”  “distributed  data,”  ‘data  interchange,”  “data 
streams,”  “collect  data,”  “RFP  models,”  “license  models,”  “coupled  models,”  “CAD 
models,”  etc. 
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